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Abstract 

The  objective  of  this  research  is  to  improve  the  available  techniques  for  discrete-event 
simulation  output  einalysis.  Discrete-event  simulations  are  dynamic  simulations  in  which  the 
system  states  change  instantaneously  at  the  occurrence  of  specified  events.  Often  the  dis¬ 
tributions  of  the  model  outputs  attain  constant  or  steady-state  characteristics  after  passing 
through  an  initial  transient  period.  The  presence  of  this  initial  transient  can  bias  estimates 
of  the  parameters  which  characterize  the  steady-state  distributions.  One  common  method 
to  prevent  bias  from  initial  data  is  to  delete  the  data  previous  to  a  selected  truncation  point. 
Currently,  only  graphical  and  heuristic  algorithms  cire  available  to  determine  the  appropriate 
initial  data  truncation  point.  This  research  investigates  the  use  of  the  Kalman  filter  to  select 
the  truncation  point  for  both  univariate  and  multivariate  simulation  output. 

Although  residual  monitoring  is  tested,  the  best  truncation-point  selection  results  are 
obtained  with  Multiple  Model  Adaptive  Estimation  (MMAE).  In  applying  the  MMAE  tech¬ 
nique,  the  unknown  parameter  space  is  discretized  and  a  bank  of  Kalman  filters  with  one 
filter  a.ssociated  with  each  discrete  point  in  parameter  space  is  initialized.  The  probability 
of  the  parameters  in  each  filter  being  correct  is  calculated  based  upon  the  filter  residuals. 
Weighting  the  filters’  estimates  by  their  probabilities  results  in  a  time-varying  MMAE  pa¬ 
rameter  estimate.  In  this  application,  the  unknown  parameter  is  the  system  mean.  The 
truncation  point  is  selected  as  the  point  when  the  time-varying  MMAE  mean  estimate  is 
within  a  small  tolerance  of  an  assumed  state-state  mean  estimate.  The  procedure  is  applied 
to  single  output  sequences  and  to  the  average  of  multiple  replications  with  excellent  results. 
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Identification  of  the 
Initial  Transient  in 
Discrete- Event  Simulation  Output 
Using  the 
Kalman  Filter 


1.  Introduction 

The  objective  of  this  research  is  to  improve  steady-state  multivariate  output  anal¬ 
ysis  techniques  for  discrete-event  simulation  through  application  of  Kalman  filters.  The 
primary  focus  is  to  determine  analytically  truncation  points  that  reduce  the  bias  from  ini¬ 
tial  conditions  in  univariate  and  multivariate  discrete-event  simulation  output  sequences  for 
steady-state  analysis.  This  introduction  provides  an  overview  of  discrete-event  simulations, 
steady-state  output  analysis,  and  Kalman  filters.  Based  on  this  foundation,  the  research 
objectives  are  outlined.  The  background  and  details  of  current  output  analysis  techniques 
are  presented  in  Chapter  II.  Chapter  III  discusses  Kalman  filters  and  system  identification. 
In  Chapters  IV  and  V,  new  methodologies  for  selecting  truncation  points  for  single  out¬ 
put  sequences  are  proposed  and  tested.  Although  the  residual  monitoring  approach  applied 
in  Chapter  IV  is  not  as  effective  a.s  the  techniques  developed  in  later  chapters,  the  model 
formulation  developed  is  applied  throughout  this  research.  In  Chapter  VI,  the  truncation- 
point  selection  is  applied  to  situations  with  multiple  replications,  and  the  methodology  is 
extended  to  multivariate  output  sequences  in  Chapter  VII.  Summary  and  recommendations 
are  presented  in  Chapter  VIII. 
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1.1  Discrete-Event  Simulation 


When  no  analytical  technique  is  known  to  investigate  a  complex  system  of  interest 
fully,  computer  simulation  often  is  used.  In  general,  the  system  under  study  is  defined  to  be 
a  collection  of  entities,  such  as  people  or  machines,  that  act  and  interact  together  toward  the 
accomplishment  of  some  logical  end  [53:3].  The  mathematical  or  logical  relationships,  which 
constitute  a  model,  describe  the  behavior  and  interactions  between  the  entities.  While  the 
model  is  an  abstraction  of  the  real  or  proposed  system,  the  model  is  designed  to  represent 
or  simulate  those  characteristics  of  interest  in  the  real  system  sufficiently  to  provide  useful 
insight.  In  a  computer  simulation,  numerical  techniques  are  used  to  conduct  experiments 
that  estimate  the  desired  characteristics  of  the  model.  Since  the  model  abstracts  the  real 
world  with  respect  to  specific  measures  of  effectiveness,  useful  inferences  about  the  real 
system  may  be  made  from  the  simulation  model’s  output. 

Computer  simulations  can  be  classified  as  deterministic  or  stochastic,  static  or  dynamic, 
and  continuous  and/or  discrete.  For  deterministic  simulations,  the  exogenous  input  variables 
are  known.  In  contrast,  stochastic  simulations  have  some  deterministic  input  parameters, 
but  they  are  driven  by  random  (actually  pseudorandom)  inputs.  For  example,  a  simulation 
of  the  partial-differential  wave  equation  would  be  a  deterministic  simulation  since  the  inputs 
are  the  known  boundary  conditions.  On  the  other  hand,  the  simulation  of  a  lottery  would 
be  a  stochastic  simulation  because  the  draw  would  be  modeled  by  a  random  variate. 

Stochastic  simulations  can  again  be  categorized  two  ways;  static  or  dynamic.  The 
simulation  system  state  is  defined  as  a  collection  of  variables  that  describe  the  system  at 
a  particular  time  relative  to  the  objectives  of  the  study  [53:3].  In  static  simulations,  the 
state  does  not  change  with  time.  Examples  of  static  stochastic  simulations  are  distribution 
sampling  and  classical  Monte  Carlo  methods  for  estimating  integrals  [103].  On  the  other 
hand,  the  state  variables  of  a  dynamic  simulation  change  over  time.  An  example  of  a  dynamic 
simulation  is  a  single-server  queue  in  which  one  of  the  simulation  state  variables  signifies 
whether  the  server  is  busy  or  idle.  This  simulation  state  variable  will  change  depending 
upon  the  entities’  interarrival  and  service  times. 
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Figure  1.  Discrete- Event  Simulation  Model 
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A  final  way  to  classify  computer  simulations  is  continuous,  discrete,  or  a  combination  of 
continuous  and  discrete.  The  state  variables  in  continuous  simulations  usually  are  described 
by  rate  or  differential  equations,  eind  these  values  are  simulated  by  approximating  the  equa¬ 
tion  solutions  over  small  intervals  of  time.  In  contrast,  the  variables  of  interest  in  discrete 
systems  change  instantaneously  at  distinct  points  in  simulated  time.  These  transition  points 
correspond  to  the  occurrences  of  particular  events.  For  exzunple,  in  a  single-server  queue, 
the  server  state  of  busy/idle  can  only  switch  either  when  an  entity  arrives  or  when  an  en¬ 
tity  completes  service.  Combined  models  using  both  continuous  and  discrete  state  variables 
exist. 

A  discrete-event  simulation,  the  topic  of  this  research,  is  computer  simulation  which  is 
stochastic,  dynamic,  and  discrete.  In  Figure  1,  a  general  depiction  is  shown.  The  vector  d 
represents  the  deterministic  input  parameters  while  c  represents  the  stream  of  concomitant 
or  control  random  variates  that  are  generated  and  used  within  the  simulation  model.  The 
model  represents  the  entities  and  their  interactions,  and  y„  represents  the  vector  of  outputs 
at  time  index  The  parameters  of  interest  €ure  the  values  which  characterize  the  distribution 
of  the  sequence  of  output  vectors  {yn} 

Queuing  theory  is  a  class  of  models  which  study  dynamic  systems  in  which  entities 
arrive  in  a  random  manner  at  a  service  facility  [79:304-364].  Since  queuing  theory  provides 
analytical  solutions  for  classes  of  systems,  these  systems  are  useful  for  simulation  research. 
Queuing  models  are  classified  by  the  distribution  of  interarrival  times,  the  distribution  of 
service  times,  and  the  number  of  servers.  Using  M  to  designate  the  exponential  (Markovian) 
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distribution  [35:597],  the  waiting  line  for  a  single  server  with  exponential  interarrival  times 
£ind  service  times  is  called  an  M/M/1  queue.  While  this  system  has  an  analytical  solution, 
this  system  also  can  be  modeled  by  means  of  a  discrete-event  simulation.  In  this  example,  the 
control  variates  c  are  the  streams  of  exponentially  distributed  interarrival  times  and  service 
times.  The  deterministic  inputs  d  would  be  the  means  of  the  exponential  distributions.  The 
model  consists  of  the  first-in-first-out  (FIFO)  service  order,  which  means  that  the  customer 
who  has  been  waiting  the  longest  gets  the  server  next.  More  complex  queues  may  have 
additional  deterministic  inputs,  such  as  maximum  queue  length,  and  the  model  may  also  be 
more  elaborate  with  rules,  such  as  balking  or  a  different  order  of  service.  Balking  means 
that  under  specified  conditions,  customers  would  leave  rather  than  enter  the  queue  and  wait 
for  service.  An  example  of  another  service  order  is  last-in-first-out  (LIFO),  which  occurs 
when  newly  arriving  orders  are  placed  on  a  stack  and  next  order  processed  is  taken  from 
the  top  of  the  stack.  The  model  is  a  discrete-event  simulation  since  the  simulation  states, 
server  busy /idle  and  number  in  the  queue,  only  change  at  either  an  entity  arrival  or  service 
completion.  In  this  case,  a  scalar  output  sequence  {t/„}  might  be  composed  of  the  waiting 
time  for  each  entity.  Another  possible  output  measure  could  be  the  queue  length  recorded 
at  periodic  intervals.  Queuing  systems  often  are  used  in  discrete-event  simulation  research 
since  queuing  theory  provides  an  analytical  steady-state  solution  for  comparison. 

Discrete-event  simulation  has  a  wide  variety  of  applications.  In  particular,  discrete- 
event  simulation  is  used  extensively  to  evaluate  systems  while  they  are  being  designed  or  to 
compare  various  operational  alternatives  without  perturbing  the  real  system.  In  fact,  it  is 
reported  as  one  of  the  most  widely  used  operations  research  tools  [35:8].  The  Air  Force,  like 
other  organizations  with  complex  problems  to  analyze,  also  uses  discrete-event  simulation. 
For  example,  the  Air  Mobility  Command  uses  the  Mobility  Analysis  Support  System,  a  large 
discrete-event  simulation  model,  to  assess  the  impact  of  proposed  policies  and  to  evaluate 
the  capabilities  of  alternative  force  structures  [78].  In  addition,  the  Defense  Science  Board 
has  recently  recommended  that  more  extensive  simulations  be  used  to  isolate,  identify,  and 
quantify  potential  problem  areas  during  weapon  system  development  and  procurement  [29]. 
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In  1990,  the  Department  of  Defense  identified  “Simulation  and  Modeling”  as  one  of  the 
top  twenty  critical  technologies  to  ensure  long-term  qualitative  superiority  of  United  States 
weapon  systems  [74]. 

Summary  articles  on  discrete-event  simulation  include  Shannon  [92],  Law  [50],  Wilson 
[103],  Kelton  and  Law  [45].  Numerous  books  have  been  published  on  discrete-event  simula¬ 
tions  including  Shannon  [93],  Fishman  [23],  Kleijnen  [47],  Banks  and  Carson  [10],  and  Law 
and  Kelton  [53].  In  addition,  several  simulation  languages,  such  as  SLAM  [77],  are  used 
throughout  the  analysis  community. 

1.2  Output  Analysis 

One  of  the  principal  objectives  of  discrete-event  simulation  is  to  estimate  the  character¬ 
istics  of  the  model  outputs  of  interest.  Since  the  simulation  outputs  are  functions  of  the  input 
random  variables,  the  outputs  are  a  sample  from  a  discrete-time  stochastic  process.  The  ob¬ 
jective  of  output  analysis  is  to  estimate  the  parameters  of  the  output  distribution.  Typical 
parameters  of  interest  are  the  mean,  the  variance,  a  specified  quantile,  or  the  probability  an 
output  variate  is  within  a  specific  interval  [103]. 

Typically  the  output  sequence  is  correlated  in  time;  hence,  the  estimation  process  is 
similar  to  time  series  analysis  where  inferences  are  made  about  a  stochastic  process  based  on 
partial  observation  [12:1].  However,  unlike  time  series  analysis,  additional  realizations  can 
be  obtained  from  separate  runs  of  the  model.  Since  the  inputs  for  discrete-event  simulations 
are  typically  streams  of  independent,  identically  distributed  (iid)  random  variables  that  are 
different  for  each  simulation  run,  each  simulation  run  or  replication  results  in  an  independent 
realization  of  the  stochastic  process.  The  goal  of  output  analysis  is  to  make  inferences  about 
the  stochastic  process  and  the  associated  distributions  based  on  realizations  of  the  simulation 
output.  However,  achieving  this  goal  may  be  limited  by  the  cost  of  computer  time  required 
to  generate  the  necessary  amount  of  simulation  output  data  [53:523-524]. 

Since  the  output  sequences  are  realizations  of  a  stochastic  process,  any  estimates  that 
are  derived  from  them  are  random  variables.  Welch  [101]  summarizes  the  output  analysis 
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process  as  follows: 


Hence  the  experimenter  must  generate  from  the  simulation  not  only  an  estimate 
(i  but  also  enough  information  about  the  probability  distribution  of  p  so  that  he 
can  be  reasonably  sure  that  it  is  close  enough  to  the  unknown  quantity  fi  for  his 
purposes.  [101:279] 

Usually,  this  is  accomplished  by  using  an  approximately  unbiased  estimator,  so  that  E[fLy]  a; 
fly,  and  estimating  the  variance  of  the  estimator  fty  from  sets  of  sample  sequences 
[101:279].  The  estimated  variance  along  with  an  assumed  sampling  distribution  are 
used  to  construct  a  confidence  interval  around  the  estimated  mean  fiy. 

Although  a  model  is  an  abstraction  of  reaJity,  in  practice,  the  validity  of  a  model  is 
assumed  in  output  analysis.  Therefore,  statistical  evaluations  of  alternatives  are  made  based 
on  the  model  output  without  considering  unmodeled  effects.  The  impact  of  this  assumption 
is  mitigated  by  the  practice  of  analysts  and  decision  makers  to  disregard  small  statistically 
significant  differences  as  not  being  practical  discriminators  between  alternatives.  Because 
of  the  abstraction  of  reality  involved  in  simulation  modeling,  discrete-event  simulation  is 
better  suited  for  relative  comparisons  between  alternatives  than  for  absolute  determination 
of  system  performance  [15]. 

Depending  upon  the  objectives  of  the  study,  two  types  of  analysis,  finite-horizon  and 
infinite-horizon,  are  typically  conducted  with  discrete-event  simulation  models.  Although 
this  research  is  limited  to  infinite-horizon  analysis,  both  types  will  be  described  here  for 
completeness. 

l.S.l  Finite- Horizon  Simulation  Analysis.  If  the  parameters  of  interest  are  for  the 
output  from  an  initial  condition  until  a  natural  conclusion,  finite-horizon  analysis  is  con¬ 
ducted.  The  initial  condition  should  be  representative  of  expected  conditions  in  the  actual 
system.  For  example,  in  a  retail  simulation,  the  initial  conditions  could  correspond  to  the 
number  of  customers  and  servers  in  the  system  when  the  simulation  begins.  A  reasonable 
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number  of  customers  may  be  zero  if  tbe  simulation  is  beginning  at  the  store’s  morning  open¬ 
ing.  The  natural  conclusion  for  a  terminating  simulation  may  be  the  closing  of  a  business, 
completion  of  a  contract,  completion  of  a  military  engagement,  or  end  of  the  planning  hori¬ 
zon.  Since  the  period  of  interest  is  from  the  initial  condition  to  the  natural  conclusion, 
the  model  is  run  through  this  simulated  time.  The  simulation  is  repeated  to  improve  the 
parameter  estimates  and  determine  their  variability. 

1.2.2  Infinite- Horizon  Simulation  Analysis.  If  steady-state  stationary  or  cyclic  out¬ 
put  distributions  are  of  interest  [53:528-531],  infinite-horizon  simulation  analysis  techniques 
are  used.  When  steady-state  stationary  output  distribution  exist,  the  output  will  attain  a 
stationary  probability  distribution  as  the  length  of  the  simulation  run  increases.  In  con¬ 
trast,  steady-state  cyclic  output  will  follow  a  time-varying  probability  distribution  as  the 
simulation  output  length  increases.  Although  simulation  models  typically  do  not  have  an 
analytical  equation  to  describe  their  output,  these  classifications  are  similar  to  the  solutions 
of  linear  differential  equations  with  time-invariant  and  time- varying  coefficients.  This  re¬ 
search  focuses  on  estimating  the  output  parameters  of  discrete-event  simulation  models  that 
attain  a  stationary  steady-state  probability  distribution. 

While  the  object  of  finite-horizon  analysis  is  to  estimate  the  output  distribution  pa¬ 
rameters  for  specified  conditions,  infinite-horizon  analysis  estimates  the  long-run  values  of 
output  distribution  parameters.  In  infinite-horizon  analysis,  particular  attention  is  paid  to 
eliminating  the  effects  of  the  model’s  start-up  conditions.  For  example,  a  logistics  system 
could  be  modeled  to  determine  the  effect  of  more  repair  facilities  on  the  expected  repair 
time.  Even  though  the  repair  facilities  may  close  every  day,  with  the  assumption  that  work 
picks  up  where  it  left  off,  the  sequence  of  repair  times  would  be  uneffected  by  the  breaks. 
Infinite-horizon  analysis  indicates  the  long  run  capacity  of  the  logistic  system  regardless  of 
how  many  failed  components  are  initially  in  the  model. 

In  infinite-horizon  analysis,  two  problems  typically  are  present.  First,  the  duration  of 
significant  effect  of  the  initial  conditions  on  the  output  is  not  known  a  priori.  Second,  classical 
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statistical  techniques  are  usually  based  on  independent  observations,  but  the  sequence  of 
observations  from  the  simulation  model  typically  are  correlated  [101:289].  A  considerable 
amount  of  research  in  this  field  has  been  devoted  to  develop  output  analysis  techniques  which 
address  these  two  problems. 

If  the  simulation  output  sequence  has  a  stationary  steady-state  distribution,  then  this 
sequence  typically  passes  through  a  transient  phase,  which  depends  upon  the  initial  con¬ 
ditions.  Thereafter,  in  the  steady-state  phase,  the  sequence  has  an  essentially  unchanging 
distribution  which  is  independent  of  the  initial  conditions.  Rather  than  the  transient  phase 
ending  at  a  particular  point,  the  behavior  of  the  sequence  gradually  may  converge  to  steady- 
state  behavior  [101:277-278].  This  convergent  behavior  depends  upon  the  particular  initial 
conditions  and  the  process  dynamics.  Since  the  transient  output  characteristics  are  not 
representative  of  the  steady-state  characteristics,  including  the  simulation  output  from  the 
beginning  of  a  run  may  bias  steady-state  estimates.  F.  *vvver,  determining  the  point  at  which 
the  output  sequence  effectively  has  atta'aed  steady-state  is  difficult. 

While  most  classical  statistic  techniques  apply  to  independent  replications,  simulation 
output  usually  is  correlated  positively  within  a  model  run.  This  correlation  reduces  the  new 
information  available  for  subsequent  output  values.  Therefore,  the  run  length  must  be  longer 
to  achieve  an  equivalent  level  of  confidence  in  parameter  estimates.  Various  techniques  exist 
to  attain  approximate  independence  or  directly  account  for  the  effects  of  the  correlation  in 
the  output. 

In  the  following  quote,  Kelton  and  Law  summarize  the  current  limitations  in  deter¬ 
mining  the  simulation  run  length  and  truncation  point  so  that  the  sample  average  of  the 
truncated  sequence.  Xml  in  their  notation,  is  a  good  estimate  of  the  mean  of  the  output 
distribution  /z. 

What  is  really  needed,  then,  is  a  generally  reliable  method  for  identifying  deletion 
amounts  I,  as  well  as  (total)  replication  length  m  such  that  Umi  is  sufficiently  close 
to  fji  to  allow  us  safely  to  use  the  Xmt  as  being  unbiased  estimators  for  /x  in  the 
context  of  a  given  inferential  goal,  e.g.  c.i.  [confidence  interval]  formulation.  [45] 
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Further  background  on  discrete-event  simulation  output  analysis  is  in  Chapter  II. 

1.3  Kalman  Filters 

The  Kalman  filter  is  an  algorithm  to  calculate  optimal  estimates  of  the  state  of  a 
stochastic  process  based  on  imperfect  observations  of  the  system  output.  Another  definition 
is  that  the  “Kalman  filter  is  simply  an  optimal  recursive  data  processing  algorithm”  [57:4]. 
In  this  definition,  optimality  can  be  established  by  various  criteria,  such  as  minimizing  any 
symmetric  cost  function.  The  term  recursive  relates  to  the  two-step  nature  of  the  computer 
program  to  propagate  an  estimate  and  then  to  update  the  estimate  with  an  imprecise  mea¬ 
surement,  repeatedly.  The  details  and  theoretical  presentation  of  the  Kalman  filter  are  given 
in  Chapter  III. 

While  the  Kalman  filter  is  used  primarily  on  engineering  applications,  statisticians  have 
implemented  the  algorithm  as  an  optimal  Bayesian  estimator  [67].  In  applying  the  Kalman 
filter  to  simulation  output,  additional  information  about  the  stochastic  process  results  in  the 
form  of  a  state-space  representation,  state  estimates,  the  state  estimate’s  covariances,  resid¬ 
uals,  and  computed  residual  covariances.  This  research  exploits  this  additional  information 
to  improve  simulation  output  analysis  techniques. 

1.4  Research  Objective 

With  the  increase  in  computer  capabilities  and  improved  simulation  languages,  larger 
and  more  complex  simulation  models  are  being  built  and  used.  Simulations  of  complex  sys¬ 
tems  may  require  large  amounts  of  computer  time  [53:523].  Generally,  these  models  produce 
an  array  of  outputs,  complicating  the  necessary  multivariate  output  nalysis.  For  example, 
in  steady-state  simulations,  the  initial  transient  data  should  be  eliminated  to  prevent  biasing 
the  estimates.  However,  in  order  to  determine  the  effective  end  of  the  transient  phase,  each 
sequence  of  output  variables  should  be  checked  for  stabilization.  The  determination  of  the 
multivariate  estimates  along  with  their  associated  confidence  region  and  the  determination 
of  sufficient  simulation  run  length  are  also  more  difficult  with  larger  complex  models.  Im- 
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proving  the  available  techniques  to  conduct  univariate  and  multivariate  output  analysis  of 
discrete-event  steady-state  simulation  is  the  general  objective  of  this  research. 

This  research  effort  focuses  on  identifying  the  effective  end  of  the  transient  phase,  or 
equivalently,  the  effective  beginning  of  steady-state  phase.  The  objective  of  this  dissertation 
is  to  develop  improved  analytical  techniques  for  determining  truncation  points  that  effectively 
reduce  the  remaining  bias  due  to  the  initial  conditions.  Techniques  are  proposed  for  both 
univariate  and  multivariate  output  sequences.  The  truncation  point  that  are  selected  by  the 
procedures  should  be  far  enough  into  the  simulation  run  to  reduce  the  bias  induced  by  the 
transient  effects,  but  not  so  far  that  data  is  deleted  unnecessarily,  increasing  the  variance  of 
the  estimates. 

1.5  Preview 

The  remainder  of  this  dissertation  is  organized  as  follows:  Chapter  II  provides  simu¬ 
lation  output  analysis  background.  Kalman  filters  and  estimation  techniques  are  described 
in  Chapter  III.  In  Chapter  IV,  a  truncation  identification  point  based  on  running  a  single 
Kalman  filter  in  conjunction  with  residual  monitoring  is  developed  and  tested.  While  the 
residual  monitoring  approach  did  not  provide  expected  results,  the  same  model  formulation, 
developed  in  Section  4.2,  is  used  throughout  this  dissertation.  Chapter  V  presents  a  different 
approach  based  on  Multiple  Model  Adaptive  Estimation  (MMAE)  with  a  bank  of  Kalman 
filters.  Many  popular  discrete-event  simulation  confidence  interval  techniques  are  based  on 
a  single  long  run  of  output  [13,  39,  52,  69,  53:551-563].  Because  of  the  run  length  used  with 
these  techniques,  the  parameter  estimates  may  be  biased  an  insignificant  amount  by  the  tran¬ 
sient.  While  the  MMAE  approach  works  on  a  single  long  run,  the  value  of  the  additional 
effort  to  truncate  data  which  induces  a  very  small  bias  may  be  questionable.  In  contrast, 
the  multiple  replication  confidence  interval  technique  uses  several  output  sequences.  Since 
each  sequence  contains  initial  transient  data,  the  determination  of  an  appropriate  truncation 
point  is  more  important  to  prevent  biased  parameter  estimates.  The  MMAE  approach  is 
applied  to  the  case  of  multiple  replications  of  univariate  sequences  in  Chapter  VI,  and  the 
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method  is  extended  to  multivariate  output  in  Chapter  VII.  The  dissertation  is  completed 
with  conclusions  and  recommendations  in  Chapter  VIII. 
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II.  Steady-State  Output  Analysis  Techniques 


2. 1  Introduction 

Simulation  output  analysis  can  be  thought  of  as  a  collection  of  analytical  techniques 
used  to  maJce  inferences  about  the  probability  distributions  of  the  outputs  from  discrete-event 
simulation  models.  These  analysis  techniques  are  used  to  make  estimates  of  parameters  which 
reflect  the  properties  of  underlying  probability  distributions  of  outputs.  Typical  parameters 
that  are  estimated  are  the  mean,  variance,  quantiles,  or  the  probability  an  output  observation 
is  within  an  interval  [101,  103]. 

Parameters  for  several  types  of  output  are  typically  of  interest.  Conway  [15]  classified 
the  types  of  output  as  based  on  either  temporary  or  permanent  entities.  An  output  based 
on  a  “temporary  entity”  is  associated  with  a  particular  entity  that  flows  through  the  system. 
For  example,  the  waiting  times  associated  with  customers  in  a  queue  are  temporary  variables 
because  each  waiting  time  is  associated  with  a  specific  customer.  The  name  temporary  is  used 
since  the  variable  only  exists  while  the  entity  is  in  the  system.  Whcxi  the  entity  completes 
service  and  is  terminated,  the  temporary  variable  no  longer  exists.  Pritsker  [77:36]  calls 
the  output  based  on  temporary  entities  “statistics  based  upon  observations” .  On  the  other 
hand,  “permanent  entities”  remain  throughout  the  simulation  run.  Examples  of  permanent 
variables  are  the  number  in  the  queue  and  busy /idle  status  of  a  server.  Permanent  variables 
change  during  the  simulation  run,  and  usually  the  variable’s  average  value  over  time  is  the 
parameter  of  interest  for  each  such  variable.  Pritsker  [77:36]  refers  to  output  which  measures 
permanent  variables  as  “statistics  based  on  time-persistent  variables”.  This  research  deals 
with  sequences  of  both  “statistics  based  upon  observations”  and  “statistics  based  upon  time- 
persistent  variables”. 

The  output  may  be  produced  at  evenly-spaced  or  unevenly-spaced  time  intervals. 
Evenly-spaced  outputs  are  typically  simulation  state  variables  that  are  measured  at  set  in¬ 
tervals  of  simulated  time.  For  example,  the  number  of  entities  in  the  queue  could  be  output 
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at  a  predetermined  rate.  Unevenly-spaced  output  are  generated  at  varying  time  intervals. 
For  example,  let  the  output  sequence  be  the  waiting  times  for  entities  processing  through 
a  system.  The  waiting  time  associated  with  each  entity  is  recorded  when  the  entity  com¬ 
pletes  service.  Since  the  entities  complete  service  at  random  intervals,  their  waiting  times 
are  observed  as  unevenly-spaced  output.  While  many  of  the  techniques  are  developed  for 
evenly-spaced  output,  they  can  be  applied  to  unevenly-spaced  data  using  an  index.  For  the 
enuties  in  the  simulation  of  a  MjMjl  queue,  their  completion  sequence  number  can  be  the 
necessary  index.  This  technique  of  using  indices  to  represent  time  is  a  common  practice  in 
time  series  analysis,  and  Fishman  [20]  applies  it  to  simulation  output  data. 

Charnes  [12:8]  classifies  multivariate  discrete-event  simulation  output  as  synchronous 
or  asynchronous.  Synchronous  output  results  when  the  entire  multivariate  output  vector  is 
produced  at  the  same  time.  In  contrast,  asynchronous  output  results  when  only  a  portion 
of  the  output  vector  is  obtained  from  each  simulation  observation.  For  example,  consider  a 
simulation  of  a  medical  clinic  that  has  three  types  of  patients  and  assume  the  vector  of  waiting 
times  by  patient  type  is  of  interest.  Since  each  patient  can  only  be  one  of  the  three  types  and 
each  patient’s  completion  produces  a  simulation  output,  asynchronous  output  results.  Seila 
[90,  91]  proposed  two  techniques  for  estimating  confidence  intervals  for  asynchronous  output. 
Although  this  research  is  limited  to  synchronous  output,  a  recommendation  for  futher  work 
addresses  asynchronous  output. 

Because  of  the  randomness  of  the  input  variates  from  run  to  run,  “a  simulation  is 
a  computer-based  statistical  sampling  experiment  [50].”  Unlike  most  experimenters,  the 
simulator  has  significantly  more  control  over  the  inputs.  Therefore,  data  collection  through 
simulation  is  very  controlled  experimentation. 

Since  the  output  is  a  realization  or  sample  of  a  stochastic  process,  most  output  analysis 
techniques  are  based  on  statistical  theory.  For  example,  numerous  techniques  have  been 
developed  and  are  in  use  to  construct  a  confidence  interval  for  the  estimate  of  the  output 
mean  [13,  39,  52,  69,  53:551-563].  Some  of  the  techniques  assist  in  a  specific  portion  of 
the  output  analysis  task.  As  an  example,  the  techniques  for  initial  data  truncation  [24,  44, 
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104,  105,  101:289-294]  axe  designed  to  remove  unrepresentative  data  at  the  beginning  of  the 
simulation  run  to  improve  the  steady-state  estimates  calculated  with  other  techniques. 

In  finite-horizon  simulation  analysis,  a  simulation  model  is  run  from  a  specified  set 
of  initial  conditions  to  a  natural  ending.  For  example,  a  simulation  of  a  ground  war  may 
begin  with  all  of  the  tanks  operational  and  continue  until  one  side  is  victorious.  Running  the 
model  repeatedly  with  different  independent  random  numbers  results  in  independent  replica¬ 
tions.  Since  the  summary  statistics  are  independent  between  simulation  runs  or  replications, 
classical  statistical  techniques  are  applied. 

In  contrast  to  finite-horizon  simulation  analysis,  infinite-horizon  or  steady-state  analy¬ 
sis  techniques  are  used  to  estimate  the  long  run  or  equilibrium  characteristics  of  the  system. 
Some  discrete-event  simulation  models  follow  a  cyclic  pattern  in  steady-state,  but  this  re- 
seeirch  addresses  only  simulation  models  with  output  that  attains  a  stationary  steady-state 
probability  distribution.  In  most  real  systems  the  characteristics  of  the  system  change  over 
time,  hence  the  stochastic  process  may  not  have  a  steady-state  distribution.  In  many  cases, 
the  changes  are  slow  enough  to  allow  “quasi-static”  modeling,  and  useful  inferences  can 
be  derived  from  a  stationary  steady-state  simulation  model.  In  addition,  for  discrete-event 
simulation  models,  the  system  characteristics  are  generally  held  constant  throughout  the 
simulation,  and  therefore,  the  model  output  sequences  often  attain  a  stationary  steady-state 
distribution  [53:530]. 

The  next  section  presents  steady-state  discrete-event  simulation  assumptions  and  char¬ 
acteristics.  The  following  section  reviews  the  literature  on  steady-state  identification  tech¬ 
niques. 

2.2  Steady-State  Assumptions  and  Characteristics 

If  a  simulation  model  output  converges  to  a  stationary  steady-state  distribution,  then 
the  probability  distribution  of  the  nth  observation  would  approach  a  limiting  distribution 
for  any  given  initial  conditions  as  n  increases,  i.e.,  lim„_oo  I  >)  — ^  where  the  i 

represents  initial  conditions  [53:525].  Since  it  is  difficult,  if  not  impossible,  to  pick  initial 
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conditions  which  are  representative  of  the  steady-state  conditions,  the  simulation  model  will 
pass  through  a  trajisient  phase.  Theoretically,  the  steady  state  or  asymptotic  distribution 
is  approached  and  never  actually  attained.  The  goal  of  steady-state  analysis  is  to  run  the 
simulation  long  enough  that  the  initial  conditions  no  longer  have  a  significant  effect  on 
the  estimated  measures  of  performance  [53:531].  After  effectively  completing  the  transient 
period  at  no,  the  output  sequence  {^n}  for  n  >  rio  constitutes  a  discrete  realization  of  a 
stochastic  process  in  which  each  yn  follows  the  steady-state  distribution.  Although  members 
of  this  sequence  are  not  independent,  they  often  approximately  constitute  a  covariance- 
stationairity  process  [53,  101].  In  addition,  many  of  the  output  analysis  techniques  implicitly 
assume  ergodicity.  The  next  two  sections  describe  the  concepts  of  covariance  stationary  and 
ergodicity. 

2.2.1  Covariance  Stationarity.  Covariance  stationarity  requires  that  the  mean  of  the 
output  sequence  fjLy  is  constant  and  the  elements  of  the  covariance  matrix  are  finite  and 
constant  through  time,  and  the  degree  of  correlation  between  random  variables  in  the  se¬ 
quence  depends  only  on  the  distance  between  the  variables  and  not  on  their  absolute  location 
[53,  57,  77].  In  particular  for  a  scalar  output  sequence  {j/n}, 

fly  =  E[y(t„)] 
al  =  E[(y(t„)  -  fiyf] 

For  times  s  and  t, 


7„(t  -  5)  =  Cov[y(t),y(5)]  =  E[(y(t)  -  liy)iy{s)  -  fiy)] 

7y(0)  =  al 
7v(0  =  7v(-*) 

Pv(0  =  7w(*)/7v(0) 


15 


where  7y(i)  is  the  autocovariance  and  p{i)  is  the  autocorrelation  at  a  lag  of  i.  Covariance 
stationarity  implies  that 


E[^(t„)]  =  fly,  a  constant  for  all  n 
E[i^(t„)^]  <  oo  and  a  constant  for  all  n 

7v(^  -  •*)  =  7v(^  -  9)  if  -  -si  =  k  -  9I 

For  multivariate  output, 

/iy  =  E[y(t„)] 

Syy  =  E[(y(t„)  -  /iy)(y(i„)  -  flyf] 

and  the  autocovariance  matrix  Fyy  is  defined 

TyyCt  —  •*)  =  E[(y(it)  —  —  fiy)^] 

ry(0)  =  Eyy 

and  covariance  stationarity  implies  that 

E[y(tri)]  =  /iy  is  a  constant  vector 

E[y(t„)y^(t„)]  is  a  finite  and  constant  matrix 
EyyC^  -  5)  =  ryy(r  -  q)  if  |t  -  s|  =  |r  -  q\ 

2.2.2  Ergodicity.  Another  important  concept  is  ergodicity,  which  Maybeck  [57]  de¬ 
fines  as  follows: 

A  process  is  ergodic  if  any  statistic  calculated  by  averaging  over  all  members  of 
the  ensemble  of  samples  at  a  fixed  time  can  be  calculated  equivalently  by  time- 
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averaging  over  any  single  representative  member  of  the  ensemble,  except  possibly 
a  single  member  out  of  a  set  of  probability  zero.  [57:144] 

The  advantage  of  a  realization  of  an  ergodic  process  is  that  the  properties  of  the  random 
variable  can  be  estimated  from  a  single  re«ilization.  Parzen  [75:73]  proves  that  the  sequence  of 
sample  means  {/ty(n),  n  =  1,^  . . . ,  oo},  is  ergodic  if  -+  0  as  n  ->  oo.  Parzen  [75:74- 
75]  further  proves  that  a  necessary  and  sufficient  condition  for  an  ergodic  process  is  that 
the  elements  of  the  autocovariance  matrices  are  bounded  and  the  covaricince,  between  the 
sample  mean  fiy{n)  and  the  leist  observation  tends  to  zero  as  n  increases.  Parzen  [77:74] 
states  that  a  sufficient  condition  for  a  covariance  stationary  process  to  be  ergodic  is  that  the 
autocovariance  function  7v(j)  tends  to  zero  as  i  increases  to  infinity.  The  importance  of  this 
result  in  the  simulation  application  is  that  tin  estimator  of  the  mean  approximately  equals 
the  process  meein  if  the  variance  of  the  estimator  goes  to  zero  as  the  run  length  increases  to 
infinity  [77:727].  Many  of  the  discrete-event  simulation  output  analysis  techniques,  such  as 
all  the  confidence  interval  approaches  based  on  a  single  run,  implicitly  assume  the  simulation 
output  is  an  ergodic  sequence. 

2.3  Output  Analysis  Issues 

Most  of  the  research  on  steady-state  simulation  analysis  has  dealt  with  some  aspect 
of  the  two  problems,  the  initial  transient  or  the  correlated  output.  Many  of  the  techniques 
can  be  categorized  as  either  multiple-run  techniques  or  single-run  techniques.  Multiple  runs 
achieve  independence  by  separate  stochastic  realizations.  However,  the  initial  transient  must 
be  deleted  from  each  of  the  initial  runs.  The  single-run  techniques  use  one  long  simulation 
run  so  that  the  transient  must  only  be  deleted  once.  However,  accounting  for  the  correlation 
in  the  output  is  a  critical  problem  for  the  single-run  techniques.  The  next  two  sections  detail 
the  transient  and  correlation  problems. 

2.3.1  Transient  Issue.  The  initial  transient  or  startup  problem  is  induced  by  initial 
conditions  which  are  not  representative  of  the  unknown  steady-state  output  distribution. 
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Conway  [15]  states  that  the  transient  components  typically  decay  geometrically  in  time  and 
convergence  to  steady-state  conditions  can  be  quite  slow.  Odoni  and  Roth  [73]  numerically 
solved  the  Kolmogorov  forward  differential  equations  to  describe  the  transient  behavior  of 
certain  queuing  systems.  They  stated, 


The  rate  at  which  a  queue  converges  to  its  steady-state  characteristics,  inde¬ 
pendently  of  the  system’s  initial  state,  eventually  becomes  (for  large  values  of 
time  t)  dominated  by  an  exponential  term  of  the  form  exp(— t/r)  where  r  is  a 
characteristic  of  the  queuing  system. 


For  their  queuing  systems,  the  characteristic  r  is  found  to  vary  with  powers  of  the  coefficients 
of  variation  of  the  interarrival  and  service  times,  and  it  is  independent  of  the  initial  conditions 
selected.  Their  finding  of  exponential  decay  to  the  steady-state  output  distribution  agrees 
with  other  researchers,  su*..!  aS  Conway  [15]. 


Once  the  output  of  a  simulation  effectively  has  attained  its  stationary  steady-state 
probability  distrloution,  each  observation  has  the  same  cumulative  density  function  Fy[(), 
regardless  of  the  initial  conditions  of  the  simulation  model.  While  the  steady-state  probabil¬ 
ity  distribution  is  attained  only  in  the  limit,  the  point  after  which  steady  state  is  effectively 
achieved  is  labeled,  using  Welch’s  notation  [101:295],  as  no-  Effectively  attained  means  that 
the  error  caused  by  assuming  the  system  is  in  steady  state  is  negligible  [15].  Therefore,  rio 
is  an  estimate  of  the  end  of  significant  effects  of  the  transient.  Thus,  E[j/„]  /  fiy  for  n  <  no 
but  analysts  assume  E[y„]  =  fiy  for  n  >  no-  If  the  mean  of  the  output  sequence  is  calculated 


using  all  the  data. 


1  ^ 

Av  =  E  S/n 

n=:l 


then  the  expected  value  does  not  equal  the  steady-state  mean;  E[/ij,]  ^  fiy.  On  the  other 
hand,  if  the  output  sequence  prior  to  tiq  is  deleted,  then  the  mean  of  the  remaining  data 
would  be  an  unbiased  estimator  of  the  steady-state  stochastic  process  mean: 


E[Av]  =  E 


1  ^ 

^0  „=„o  +  l 


E  E[T/n]  =  Mv 

’^On^no  +  l 
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For  this  reason,  a  current  practice  is  to  delete  or  truncate  the  beginning  portion  of  the 
output  sequence.  The  difficulty  arises  in  determining  how  many  of  the  initial  observations 
to  delete.  Not  deleting  enough  initial  data  may  bias  the  resulting  steady-state  estimates, 
whereas  truncating  too  much  data  unnecessarily  increases  the  variance  of  the  steady-state 
estimates. 

While  the  steady-state  values  do  not  depend  on  the  initial  conditions,  the  length  and 
rate  of  convergence  do  depend  on  the  initial  conditions  for  discrete-event  simulation  models 
[53:551].  Numerous  researchers  [43,  71,  53:551]  have  investigated  the  effects  of  various  initial 
conditions. 

Madansky  [56]  examines  (M/Af/1)  queues  with  no  data  deletion.  He  shows  that  if 
the  objective  is  to  minimize  mean  square  error  (MSB)  in  the  estimate  of  the  number  in 
the  system,  the  mode  and  not  the  steady-state  mean  is  the  optimal  number  of  queued 
observations  for  an  initial  condition.  In  addition,  increasing  simulation  run  length  is  more 
effective  than  adding  replications  in  reducing  the  MSB  [56].  Bxtending  Madansky  results  to 
the  case  where  deletion  is  allowed  is  not  obvious. 

Kelton  and  Law  [46]  analyze  the  transient  behavior  of  different  queues.  Their  findings 
indicate  that,  for  systems  with  several  servers,  starting  empty  and  idle  with  a  build-up 
to  steady-state  is  much  slower  than  beginning  overcongested  and  drawing-down  to  steady 
state.  Initial  conditions  can  be  specified  based  on  the  output  of  pilot  runs.  While  most  of  the 
literature  favors  the  mode  of  the  number  in  the  system  as  the  best  initial  condition,  Kelton 
and  Law  [46]  prefer  the  mean  or  greater. 

Other  initialization  routines  have  been  proposed.  Law  and  Kelton  [53]  credit  Glynn 
for  suggesting  a  one-time  pass  through  the  transient  to  develop  starting  conditions.  Kelton 
[43]  proposes  a  random  initialization  which  reduces  the  severity  and  duration  of  the  initial 
transient  period. 

However,  while  these  papers  demonstrate  the  significant  impact  of  the  initial  condi¬ 
tions,  in  actual  practice,  these  techniques  are  difficult  to  implement.  First,  real-world  multi- 
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variate  simulations  have  many  queues  and  factors  which  would  be  required  to  be  initialized. 
In  addition,  many  of  these  queues  are  maintained  internal  to  the  simulation  model,  and  the 
corresponding  queue  statistics  are  not  included  in  the  output.  Therefore,  the  best  initial 
conditions  for  these  queues  could  not  be  estimated  from  the  output  of  previous  runs  without 
modifying  the  model’s  output  routines.  Regardless,  since  setting  specific  initial  conditions 
shortens,  but  does  not  eliminate  the  transient  [15],  an  improved  algorithm  to  identify  the 
end  of  significant  effects  of  the  transient  is  worthwhile. 

2.3.S  Correlation  Issue.  Besides  dealing  with  the  problem  of  the  initial  transient,  we 
must  also  contend  with  the  problem  of  correlated  output.  The  effect  of  this  correlation  is 
the  difficulty,  not  only  in  estimating  the  stochastic  process  variance,  but  also  in  estimating 
the  variance  of  estimators.  For  example,  the  variance  of  the  sample  mean  is  below: 

<  =  ^  {E"  .  <>1  +  2  Cov(j,,!,*j} 

Usually  discrete-event  simulation  output  is  correlated  positively.  Ignoring  the  positive  cor¬ 
relation  results  in  underestimating  the  variance  of  the  estimator  fiy  and  the  corresponding 
confidence  interval  being  too  small. 

2-4  Steady-State  Identification 

Since  the  run  length  of  any  simulation  is  finite,  the  true  steady-state  or  asymptotic 
distribution  is  never  really  achieved.  However,  for  the  purposes  of  analysis,  steady-state  is 
considered  to  have  begun  when  the  initial  conditions  no  longer  have  an  effective  impact  on 
the  distribution  of  the  output  sequence.  This  point  is  labeled  as  no  using  Welch’s  notation 
[101:295]. 

The  extent  of  the  transient  phase  is  difficult  to  determine.  Law  and  Kelton  [53:527] 
indicate  that  the  sample  mean  of  the  distribution  of  waiting  times  for  M/M/1  queue  with 
traffic  intensity  p  of  0.8  is  very  near  the  theoretical  steady-state  level  within  500  customers. 
Eilon  and  Chowdhury  [19]  report  that  for  a  simulation  of  an  M/M/1  queue  with  high  traffic 
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intensity,  a  substantial  difference  between  the  theoretical  and  the  estimated  steady-state 
mean  waiting  time  may  exist  for  a  batch  of  1,000  customers,  even  after  20,000  customers. 
While  Eilon  and  Chowdhury  attribute  this  to  initialization  bias,  it  may  also  be  induced  by 
process  variability  and  high  autocorrelation. 

Steady  state  is  attained  when  the  cumulative  density  function  of  the  output  sequence 
Fy{()  stabilizes.  Welch  [101;290]  proposes  that  the  most  straightforward  approach  to  identify 
steady  state  would  be  to  estimate  Fy^(()  based  on  for  each  n,  but  he  admits  the  computing 
requirements  would  be  impractical.  Theoretically,  many  of  the  techniques  would  work  with 
higher  moments,  such  as  convergence  of  the  output  sequence  variance;  however,  the  increased 
variance  in  estimating  higher  moments  makes  detection  of  the  stabilization  more  difficult. 
Therefore,  although  stabilization  of  the  mean  is  a  necessary,  but  not  a  sufficient  condition  for 
steady  state,  the  widely  applied  techniques  are  based  on  mean  estim.ates  stabilizing  [53:545- 
551]. 

Before  discussing  the  techniques  for  estimating  no  that  have  been  proposed  in  the 
literature,  the  next  section  examines  evaluation  criteria.  The  following  section  describes 
different  approaches  to  select  a  truncation  point  to  eliminate  the  biased  initial  data  along  with 
published  comparisons.  An  example  of  a  simulation  model  of  an  M/M/1  is  presented.  The 
last  section  of  the  chapter  summarizes  the  requirements  for  a  reliable  and  useful  truncation 
point  selection  technique. 

2.4-1  Truncation  Point  Evaluation  Criteria.  This  section  reviews  criteria  that  may 
be  applied  to  evaluate  truncation  point  selection  algorithms.  The  algorithms  are  generally 
evaluated  in  terms  of  the  effectiveness  of  the  truncated  sequences  to  make  correct  infer¬ 
ences  about  a  known  parameter.  The  earliest  criterion  considered  in  the  literature  is  mean 
squared  error  (MSE)  of  the  mean  estimator  for  the  truncated  sequence.  Later  researchers 
[44,  52,  104,  105]  have  recommended  using  confidence  interval  widths  and  coverage  rates 
for  evaluating  the  effectiveness  of  truncation  sequences.  Therefore,  the  confidence  interval 
construction  techniques  are  described  in  this  section.  The  first  confidence  interval  technique 
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addressed  is  the  method  of  replications,  which  is  appropriate  when  multiple  simulation  runs 
are  conducted  with  univariate  or  multivariate  output.  Besides  the  method  of  replications, 
confidence  interval  construction  techniques  designed  for  a  single  simulation  run,  nonoverlap¬ 
ping  batch  means  (NOBM)  and  overlapping  batch  means  (OBM),  are  also  described.  Since 
both  NOBM  and  OBM  require  that  a  batch  size  be  specified,  Fishman’s  method  to  select 
a  batch  size  [22]  is  included  in  the  discussion.  After  reviewing  the  confidence  interval  tech¬ 
niques  that  may  be  used  to  evaluate  the  effectiveness  of  truncation  sequences,  Schruben’s 
technique  to  test  for  initial  bias  in  a  truncated  sequence  [87]  is  described.  The  performance 
measures  of  MSE,  confidence  interval  coverage  and  half  widths,  and  the  percentage  of  trun¬ 
cated  sequences  passing  Schruben’s  initial  bias  test  are  described  in  this  section  and  applied 
in  Monte  Carlo  analyses  in  later  chapters. 

Truncation  points  are  selected  to  delete  data  at  the  beginning  of  the  simulation  which  is 
not  representative  of  the  steady-state  values.  Not  deleting  enough  data  biases  the  resulting 
estimates,  whereas  deleting  too  much  data  increases  the  variance  of  the  initial  parameter 
estimator  [53].  The  inverse  effect  of  sample  size  can  be  seen  through  the  derivation  for 
the  variance  of  the  estimated  mean,  shown  in  Equation  (1).  This  derivation  assumes  a 
covariance-stationarity  process.  Therefore,  the  autocovariance  jy{t  ~  s)  =  Cov[T/(t),  ?/(s)]  is 
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constajit  for  einy  set  interval  between  t  and  s. 


=  -  /i„)EjLife  -  th)] 

=  ^  E" ,  E" .  E  [(».  -  -  ^.)1 

=  ^  E£.  ZU  i.(‘  -  i) 

=  i^EiL“-V+.(Af-W)7.(A) 

=  *e£--V«(i-.‘^)7»(M 
=  Et-V„  (l  -  1!)  /-.(A) 

=  #  Et\«  (l  -  f)  />.(A) 


or  equivalently, 


(1) 


Assuming  that  the  autocorrelation  decreases  sufficiently  fast  as  the  lag  increases,  lim;i_oo  Py{h) 
0,  then  for  large  N, 

-2  r  oo  1  1  r  oo 

E  ^v(^)  =  jy  E  (2) 

/l=  — OO  /l=:  — oo 

Since  truncating  the  data  reduces  the  sample  size  to  N'  =  N  —  no,  the  variance  of  the  mean 
estimator  increases.  In  other  words,  while  initial  data  truncation  reduces  the  bias,  it  also 
increases  the  variance  of  the  mean  estimate. 

In  order  to  account  for  the  tradeoff  between  bias  reduction  at  the  expense  of  variance 
induction,  numerous  researchers  [20,  21,  14,  100]  used  the  mean  square  error  (MSE).  For  a 
point  estimator  9  of  6,  the  variance  is  cr|  =  E[(0  —  6Y\  and  the  bias  is  B{9)  =  E[5]  —  6. 
Therefore, 

MSE(0)  =  E[(0  -  9f] 

=  a]  +  B{ef 

Since  the  MSE  of  an  estimator  is  the  variance  plus  the  squared  bias,  it  appears  to  be  an 
excellent  criterion  to  determine  the  correct  tradeoff  between  increased  variance  and  reduced 
bias. 
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Fishmaa  [21]  analyzes  the  effectiveness  of  truncation  on  data  from  a  first  order  autore¬ 
gressive  AR(1)  model,  (y„  —  fiy)  =  ct{yn-i  —  M»)  +  with  (—1  <  a  <  1).  The  contribution 
of  the  initial  condition  decreases  by  a”  as  the  observation  number  n  increases.  Fishman 
compares  no  data  deletion  and  a  deletion  point  no  such  that  the  initial  value  yi  contributed 
less  than  a  predetermined  percent  to  For  a  range  of  a’s  and  a  range  of  truncation 

points  based  on  various  percentages,  Fishman  shows  analytically  that  deleting  the  initial 
transient  reduces  bias,  but  may  significantly  increase  variance.  In  fact,  in  terms  of  MSE, 
Fishman  concludes  that  truncation  may  not  always  be  advisable.  Turnquist  and  Sussman 
[100]  extend  Fishman’s  work  to  single  server  queues  with  various  traffic  intensities  and  found 
that,  to  minimize  MSE,  truncation  may  not  be  worthwhile.  Wilson  and  Pritsker  [105]  report 
that  Blomqvist  shows  MSE  is  minimized  by  no  data  deletion  for  a  special  class  of  queuing 
systems.  For  Markov  processes,  Cheng  [14]  makes  an  argument  for  one  continuous  run  versus 
replications  based  on  MSE  under  the  assumptions  of  no  initial  truncation  and  the  bias  being 
a  monotonically  decreasing  function  of  sample  size. 

These  researchers  use  MSE  of  the  mean  estimator  of  the  truncated  sequence  as  a  cri¬ 
terion  to  evaluate  truncation  point  selection  algorithms.  Later  researchers  [44,  52,  104,  105] 
consider  other  criteria,  in  particular,  confidence  interval  half  widths  and  their  associated 
coverage  rates.  The  next  part  of  this  section  describes  three  confidence  interval  construction 
techniques,  the  method  of  replications,  nonoverlapping  batch  means  (NOBM)  and  overlap¬ 
ping  batch  means  (OBM),  that  are  applied  in  the  Monte  Carlo  analyses. 

The  objective  of  simulation  analysis  is  to  make  statistical  inferences  about  the  distribu¬ 
tions  of  output  sequences.  These  inferences  are  made  typically  in  terms  of  a  point  estimator 
and  an  associated  confidence  interval  for  a  prescribed  level  of  confidence.  In  these  procedures, 
a  tradeoff  exists  between  the  width  of  the  confidence  interval  and  the  level  of  confidence. 
Since  the  interval  typically  is  choosen  to  be  symmetric  about  the  point  estimator,  the  half 
width  of  the  confidence  interval  generally  is  used  for  comparisons  between  techniques.  These 
same  criteria  can  be  used  to  evaluate  the  effect  of  truncating  output  sequences  at  different 
points  to  eliminate  the  initial  transients. 


24 


In  Monte  Carlo  analyses,  the  percentage  of  times  that  confidence  intervals  contain 
a  known  steady-state  parameter  is  called  the  coverage  rate.  Nominal  coverage  rates  are 
reflected  by  the  level  of  confidence,  1  —  a,  where  a  is  the  theoretical  proportion  of  confidence 
intervals  that  do  not  contain  the  true  parameter.  Law  and  Kelton  [53:561-562]  report  that 
actual  coverage  rates  have  been  found  to  be  considerably  less  than  the  expected  coverage 
rates.  Wilson  and  Pritsker  [104,  105]  say  the  actual  coverage  attained  is  a  more  important 
criterion  than  MSE  to  the  objective  of  simulation  output  analysis.  A  bias  in  the  point 
estimator  would  cause  the  confidence  intervals  to  contain  the  true  parameter  significantly 
less  often  than  the  nominal  level  of  the  specified  confidence  interval.  Therefore,  Wilson  and 
Pritsker  [104]  propose  a  method  to  fix  the  confidence  interval  width  for  various  truncation 
points  and  initial  conditions  in  order  to  compare  the  alternative  deletion  techniques  in  terms 
of  their  actual  coverage  rates.  Kelton  and  Law  [44]  state,  “by  looking  at  criteria  other  than 
point  estimator  mean-square-error  (in  particular,  true  c.i.  [confidence  interval]  coverage 
probability  and  expected  width),  deletion  cein  improve  statistical  validity  with  only  minor 
losses  in  efficiency.”  Kelton  and  Law  [45]  studied  the  effects  of  various  deletion  amounts 
and  number  of  replications  for  an  AR(1)  process,  and  they  found  that  deletion  improves  the 
confidence  interval  coverage  rates  to  acceptable  levels  without  substantially  widening  the 
confidence  interval  half  widths. 

Confidence  intervals  for  parameters  of  discrete-event  simulation  output  distributions 
can  be  constructed  in  a  number  of  ways.  If  multiple  simulation  runs  are  made,  the  method 
of  replications  is  used.  Since  each  replication  is  initialized  with  a  different  pseudorandom 
number  generator  seed,  the  sample  output  sequences  are  independent.  Let  ym{N)  be  the 
mean  of  the  mth  replication  with  N  sequential  observations.  Based  on  M  replications,  the 
mean  estimate  is 

(m)  S 

with  sample  variance  of 

^2  _  ^  (Vm(N)  — 

m-1 

Tn=i 
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A  100(1  —  a)  percent  confidence  interval  is  constructed  using 


Ay  d: 


(3) 


where  td,i-a/2  is  the  upper  1  —  a/2  critical  value  for  the  Student’s  t  distribution  with  degrees 
of  freedom  d  =  M  —  1. 

The  method  of  replications  can  be  extended  to  multivariate  output  to  determine  a 
confidence  ellipsoid  and  its  associated  volume.  For  M  replications  with  N  observations 
each  with  S  responses,  let  ym.si^)  be  the  mean  response  of  the  mth  replication  for  the  sth 
response.  Let  fty  be  the  vector  of  average  responses  across  replications,  and  let  the  sample 
covariance  matrix  Eyy  =  E^=i(ym(A’)  -  Ay)(ym(iV)  -  /iy)^  .  Under  the  assumption 

that  the  ym{N)  are  distributed  normcilly  with  mean  fiy  and  covariance  Syy,  the  confidence 
region,  as  shown  in  Rubinstein  and  Marcus  [80],  is  derived  from: 

Pr  |M(/ty  -  /iy)^Sy^(piy  -  /iy)  ^  =  1  -  a  (4) 


where  Fs,M-s,i-a  denotes  the  (1  —  a)  quantile  of  the  F  distribution  with  S  and  M  —  S 
degrees  of  freedom.  The  associated  volume  of  the  confidence  region  is  calculated  via 


V  =  ic(S)|E„|5 


S(M  -  1) 
M(M  -  S) 


S 

7 


Fs,M-S-,l-a 


(5) 


where  C{S)  =  and  F  is  the  gamma  function. 

If  only  one  long  univariate  simulation  run  is  conducted,  two  commonly  applied  confi¬ 
dence  interval  construction  techniques  are  nonoverlapping  batch  means  (NOBM)  and  over¬ 
lapping  batch  means  (OBM).  NOBM  is  described  in  almost  every  simulation  textbook 
[23,  51,  53,  77].  NOBM  is  similar  to  independent  replications  except  that  the  replications’ 
means  are  replaced  with  batch  means.  Each  of  the  M  batch  means  ym{B)  is  based  on  B  se¬ 
quential  observations.  If  the  batch  size  B  is  sufficiently  large,  the  batch  mean  estimators  are 
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approximately  independent  [101].  With  the  batch  means  replacing  the  replications’  means, 
the  confidence  interval  is  constructed  in  the  exact  manner  as  the  method  of  replications, 
Equation  (3).  Meketon  and  Schmeiser  [69]  propose  that  it  is  the  number  of  batches,  not 
their  independence,  which  is  critical.  By  overlapping  the  batches  in  OBM,  they  state  that 
the  asymptotic  variance  can  be  reduced  to  |  of  NOBM  variance  when  using  the  same  batch 
size.  Schmeiser  and  Song  [82]  published  FORTRAN  codes  for  both  NOBM  and  OBM. 

To  use  either  NOBM  or  OBM,  an  appropriate  batch  size  must  be  determined.  Fishman 
[22,  23:238-241]  proposes  a  methodology  to  determine  batch  size.  Fishman’s  algorithm  is 
based  on  the  correlation  between  the  batches  calculated  with 


2^  M 

where  Av  =  ^ 


(6) 


If  the  batch  means  are  independent  and  normally-distributed,  with  as  few  as  eight  batches, 
Cat  is  approximately  normally-distributed  with  a  mean  of  zero  and  a  variance  of 
Based  on  Cm,  a  hypothesis  test  for  statistical  independence  can  be  tested.  In  this  research, 
if  Cm  is  outside  the  90  percent  confidence  interval  centered  at  zero,  the  null  hypothesis  of 
independence  is  rejected.  Fishman’s  algorithm  begins  with  a  batch  size  of  one,  B  =  1  with 
M  =  N,  and  doubles  the  batch  size  until  the  null  hypothesis  is  not  rejected.  If  the  batch 
size  B  increases  until  there  are  less  than  eight  batches,  M  <  8,  the  algorithm  fails  to  find 
an  appropriate  batch  size. 

Since  recent  researchers  recommend  confidence  interval  coverage  and  half  widths,  rather 
than  MSE,  to  evaluate  the  effectiveness  of  truncation  point  selection  algorithm,  this  research 
employs  those  criteria.  Specifically,  the  coverage  rates  and  average  half  widths  for  the  trun¬ 
cated  sequences  are  reported.  For  univariate  single  run  Monte  Carlo  analyses,  the  confidence 
interval  techniques  of  NOBM  and  OBM  using  Fishman’s  batch  size  algorithm  are  applied. 
For  multiple  runs  Monte  Carlo  analyses,  the  method  of  replications  coverage  rates  and  av¬ 
erage  half  widths  are  reported.  In  addition,  Schruben’s  initial  bias  test  [87]  is  used  as  a 
criterion  for  univariate  sequences.  The  next  part  of  this  section  describes  Schruben’s  test. 
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For  a  univariate  sequence,  Schruben  [87]  uses  the  concept  of  a  Brownian  bridge  [75]  to 
develop  a  statistical  test  to  determine  if  initialization  bieis  remains  in  the  truncated  output 
sequence.  The  development  of  Schruben’s  technique  starts  by  describing  the  output  sequence 
{yn}  as  the  sum  of  the  mean  /iy(tn)  and  a  noise  term  rj{tn) 

yiu)  =  fiyitn)  +  viin)  forn  =  1 , 2, . . . ,  JV  (7) 

In  steady  state,  Hy(tn)  =  fJ-y  and  E[r7(t„)]  =  0.  If  initialization  bias  is  present,  then  fiy{tn) 
changes  in  the  sequence.  Schruben  [87]  assumes  that  the  noise  sequence  {^/(tn)}  is  stationary 
and  (^-mixing.  The  property  ^-mixing  intuitively  means  that  “distant  future  behavior  of  the 
process  is  almost  independent  of  the  present  or  past  behavior  of  the  process  [87].” 

Thinking  of  the  output  sequence  as  a  signal  Hy[tn)  plus  noise  ^/(tn),  Schruben  developes 
a  trajisformation  that  converts  the  noise  to  a  Brownian  bridge  process.  Schruben  summarizes 
a  Brownian  bridge  as  follows: 

The  limiting  stochastic  process  used  is  a  standard  Brownian  bridge 

t  e.,  Brownian  motion  on  the  unit  interval  conditioned  to  start  and 

return  to  zero.  /?<  here  is  the  process  analogue  to  the  standard  normal  random 
variable  used  in  applications  of  the  classical  central  limit  theorem.  The  Brownian 
bridge  process  has  continuous  sample  paths  and  four  notable  properties: 

1.  =  =0, 

2.  E[y3t]  =  0;  0  <  t  <  1, 

3.  Cov(^tj,/?tJ  =  min(ti,t2)(l  -  max(ti,t2)),  and 

4.  Sets  of  fit  have  a  jointly  normal  distribution.  [87] 

In  his  paper,  Schruben  shows  that  under  the  assumption  that  the  'q{tn)  are  independent 
identically  distributed  random  variables,  the  following  transformation, 

D  _  ^\N^n=lV{^n)  -  ]^Y,n=lV{^n)]  -  , 

tfk/N  - - 7= -  tor  A:  =  1, 2, . . . ,  A' 

y/Naj, 
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meets  the  first  three  criteria  of  a  Brownian  bridge  process.  The  last  criterion  is  argued  in 
the  limit. 


Under  the  assumption  of  no  initialization  bia.s,  the  transformed  sequence  can  be  mod¬ 
eled  as  a  Brownian  bridge  process.  However,  an  initialization  bias  caused  by  My(tn)  ^  My 
induces  a  non-constant  signal.  This  signal  causes  the  transformed  sequence  to  have  an  un- 
usueilly  large  value.  Under  the  assumption  of  no  signal,  the  transformed  sequence  of  noise 
is  a  Brownian  bridge  process.  Since  a  Brownian  bridge  process  follows  a  jointly  normal  dis¬ 
tribution,  a  transformed  sequence  without  noise  should  follow  a  normal  distribution  at  each 
observation.  The  presence  of  a  non-constant  signal,  which  corresponds  to  initialization  bias, 
is  apparent  as  a  statistical  outlier. 

The  actual  test  for  initialization  bias  is  based  on  the  null  hypothesis  that  no  bias  is 
present.  This  technique  finds  the  maximum  value  of  a  transformed  sequence. 


i  =  max 


N  ^n=l  Vn  f  X^n=:l 
y/N 


where  i  =  1,2,. ..  ,N 


(8) 


Let  i  be  the  index  of  the  observation  which  is  the  maximum  value,  s. 
Schruben  shows  that 

Nali{l  -  i) 


Letting  t  =  jf. 


(9) 


has  approximately  a  distribution  with  3  degrees  of  freedom  when  no  initialization  bias  is 
present.  However,  to  calculate  x  requires  the  true  variance  of  the  mean  estimator  Since 
the  ratio  of  two  approximately  independent  x^  random  variables  follows  approximately  an 
F  distribution  [36:220],  if  d?  is  estimated  with  d  degrees  of  freedom, 


has  approximately  an  F  distribution  with  3  and  d  degrees  of  freedom.  Schruben  applies 
Fishman’s  [23:247-255]  pth-order  autoregressive,  AR(p),  formulation  to  estimate  the  process 
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varieince  and  associated  degrees  of  freedom.  Fishman  derives  an  estimate  of  as 


^1  = 


- 


where  Qd  is  an  estimate  of  the  variance  of  the  zero-mean  normally-distributed  noise  term  of 
the  AR(p)  process.  Degrees  of  freedom  d  for  this  estimator  are 


n(l  -  -  <^2 - (^p) 

2p-2E^=x(p-2s)^. 


(12) 


Schruben  recommends  estimating  using  the  last  half  of  the  data  to  prevent  a  bias  from 
the  initial  data.  An  initialization  bias  caused  by  Pv(tn)  ^  Py  induces  a  non-constant  signal 
and  is  apparent  as  a  statistical  outlier.  Based  on  the  same  concepts,  Schruben,  Singh,  and 
Tierney  [88]  develope  more  powerful  tests  based  on  a  priori  knowledge  about  the  transient 
behavior. 

This  completes  the  literature  review  on  truncation  point  evaluation  criteria.  For  eval¬ 
uating  the  potential  for  future  applications  of  a  truncation  technique,  a  good  set  of  criteria 
measures  the  quality  of  confidence  intervals  constructed  with  the  truncated  sequence.  The 
confidence  interval  quality  is  measured  by  the  average  confidence  interval  widths  and  re¬ 
alized  coverage  rate.  If  the  initial  data  bias  is  removed,  the  coverage  rate  should  be  very 
near  the  nomimal  rate.  Excessive  deletion  is  apparent  at  wide  average  confidence  interval 
widths.  Another  potential  criterion  is  the  pass  rate  of  Schruben’s  test  for  initialization  bias 
detection,  a  “state-of-the-art”  approach. 


2.4-2  Truncation  Approaches.  Numerous  techniques  to  select  the  deletion  or  trun¬ 
cation  point  no  are  proposed  in  the  literature.  This  section  reviews  the  techniques  based 
on  Gafarian,  Ancker  and  Morisaku’s  [24]  survey  paper  and  Wilson  and  Pritsker’s  [104,  105] 
survey.  Both  papers  were  published  in  1978.  Since  these  surveys,  Kelton  and  Law  [44]  and 
Welch  [101]  have  proposed  two  truncation  point  selection  techniques.  Following  the  descrip¬ 
tion  of  these  methodologies,  Schruben’s  technique  for  truncating  multivariate  sequences  is 
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reviewed. 


Gafarian,  Ancker,  and  Morisaku  [24]  conduct  a  comprehensive  study  of  the  commonly 
used  techniques  to  select  the  deletion  point.  They  evaluate  the  following  five  techniques: 

Conway’s  Rule  Truncate  the  series  until  the  remaining  first  value  is  neither  the  minimum 
nor  majcimum  of  the  remaining  data. 

Conway’s  Rule  (Modified)  Find  the  first  observation  that  is  not  the  minimum  or  maxi¬ 
mum  of  the  earlier  rejected  data  values. 

Crossing-of-the-Mean  Rule  As  proposed  by  Fishman,  a  running  cumulative  mean  is 
maintained.  When  the  number  of  times  the  next  data  value  crosses  the  cumulative 
mean  (switches  from  being  above  to  below  or  vise-versa)  reaches  a  specified  number, 
select  that  as  the  truncation  point. 

Cumulative-Mean  Rule  Using  a  set  of  exploratory  simulation  runs,  the  grand  cumulative 
mean  is  plotted.  The  point  of  stabilization  is  selected  at  the  truncation  point. 

Gordon’s  Rule  Since  the  variance  of  the  mean  estimate  should  be  inversely  proportional 
to  the  sample  size,  cr^  oc  Gordon  recommends  calculating  and  plotting  on  a 
log-log  graph  with  increasing  increments  of  N.  The  N  for  which  the  graph  begins 
following  a  straight  line  with  slope  of  minus  one  half  is  selected  as  the  beginning  of 
steady-state. 

The  five  techniques  are  compared  based  on  accuracy  and  precision  of  selecting  the  “true” 
truncation  point.  The  “true”  truncation  point  is  determined  by  using  the  theoretical  expecta¬ 
tion  of  the  waiting  time  for  observations  proceeding  through  queues.  This  “true”  truncation 
point  is  selected  as  the  observation  with  a  waiting  time  within  five  percent  of  the  steady-state 
value. 

After  empirically  testing  the  rules  with  Monte  Carlo  analysis  for  a  variety  of  queuing 
systems,  Gafarian,  Ancker,  and  Morisaku  [24],  state,  “none  of  the  five  rules  is  satisfactory 
and  that  they  should  not  be  recommended  for  use  by  practitioners.” 
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Wilson  and  Pritsker  [104,  105]  survey  the  various  simulation  start-up  techniques.  Only 
the  two  additional  techniques  in  their  summary  are  presented  here. 


Fishman  [20]  developes  an  estimate  of  no  using  the  comparison  between  correlated 
observations  and  independent  observations  to  achieve  an  equivalent  reduction  in  the  vari¬ 
ance  of  the  same  mean.  The  estimate  is  based  on  the  following  result  which  is  derived  in 
Equation  (2); 


V 

N 


S  Py(^) 


k=-oo 


For  independent  observations, 


and 


p(k) 


1  for  A:  =  0 

< 

0  for  A:  ^  0 


a?  = 


^  ^independent 


Since  an  independent  observation  by  definition  is  independent  of  the  initial  conditions,  set 
^independent  =  1  ^nd  equate  the  two  equations  for  the  variance  of  the  mean  estimator; 


k=  —  oo 


Solving  for  N  results  in  a  point  in  the  data  sequence  independent  of  the  initial  conditions. 
Therefore,  the  estimated  truncation  point  is  =  SfcL-oo /^»(^)  ~  1-  The  difficulty  with 
this  procedure  is  that  estimating  the  correlation  coefficients  is  difficult.  For  finite-order 
autoregressive  (AR)  processes,  Fishman  [23]  expresses  the  truncation  point  in  terms  of  the 
autoregressive  parameters. 


Wilson  and  Pritsker  [105]  st?-te  that  Schriber  [84]  proposes  the  truncation  point  as  the 
beginning  of  a  set  number  of  batches  for  which  all  of  the  batch  means  fall  within  a  small 
interval.  Wilson  and  Pritsker  tested  these  techniques  and  conclude  that  the  truncation 
rules  of  thumb  “are  very  sensitive  to  parameter  misspecification,  and  their  use  can  result  in 
excessive  truncation.” 
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In  1983,  Kelton  and  Law  [44]  developed  a  simulation  run  length  determination  and 
truncation  point  selection  approach  based  on  linear  regression.  The  estimated  mean  for  M 
replications  and  small  batches  of  size  B  over  the  time  index  n,  B,n),  is  modeled  as 

the  population  mean  /Xy  plus  a  zero-mean  noise  term  T){tn): 

fiy{M,B,n)  =  (13) 

Under  the  assumption  that  E[i/(<„)]  converges  monotonically  to  fiy,  their  method  starts  at 
the  end  of  the  data,  fitting  regressions  on  blocks  of  data,  until  the  hypothesis  of  zero  slope  is 
rejected.  The  size  of  the  block  of  data  is  maintained  by  removing  data  at  the  end.  Batching 
is  employed  to  ensure  normally-distributed  disturbances  and  is  not  intended  to  eliminate 
autocorrelation  in  the  noise  term.  Hence,  generalized  least  squares,  rather  than  ordinary 
least  squares,  is  applied.  They  desired  a  conservative  approach  because,  to  them,  reducing 
the  bias  is  more  important  than  minimizing  the  deletion  amounts. 

In  Kelton  and  Law’s  algorithm,  M  simulation  replications  are  each  run  for  a  relatively 
short  duration.  The  averages  across  replications  are  calculated,  and  the  sequence  of  averages 
is  batched.  A  regression  line  is  fit  to  the  last  half  of  the  batch  means  and  is  tested  for 
zero  slope.  If  the  slope  is  nonzero,  each  of  the  M  simulation  replications  is  continued  to 
increase  the  length  of  their  output  sequence.  After  increasing  the  run  lengths,  the  procedure 
is  applied  again.  If  the  slope  tests  to  be  zero,  the  simulation  replication  lengths  N  are  fixed 
at  their  current  lengths.  Regression  lines  are  fit  to  earlier  segments  of  data  until  a  nonzero 
slope  is  encountered.  The  point  of  the  first  nonzero  slope  becomes  the  truncation  point  no. 

Kelton  and  Law’s  algorithm  presents  three  difficulties.  First,  continuing  simulation  run 
lengths  after  stopping  the  model  may  be  difficult  in  applications.  To  restart  the  replications, 
the  entire  simulation  state  vector  must  be  recorded  and  the  model  must  be  reset  to  these 
conditions  prior  to  restarting.  In  any  relatively  complex  simulation  model,  this  restarting  is 
difficult  to  implement.  In  addition,  besides  the  difficulting  in  restarting  the  models,  the  algo¬ 
rithm  requires  generalized  least  squares,  which  allows  for  correlated  data,  to  be  implemented. 
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Second,  the  detailed  procedure  requires  the  analyst  to  specify  nine  parameters.  These  nine 
parameters  are  the  number  of  replications,  the  initial  simulation  run  lengths,  the  amount  of 
increment  increases  to  the  run  lengths,  the  maximum  run  length,  the  number  of  batches,  the 
minimum  and  maximum  deletion  portions,  the  significance  level  for  the  test  for  zero  slope, 
and  the  mciximum  number  of  segments  over  which  a  fit  is  made.  Determining  appropriate 
values  for  these  parameters  may  prove  difficult.  Finally,  Kelton  and  Law’s  algorithm  as¬ 
sumes  monotonic  convergence  to  the  steady-state  mean.  They  recommend  further  work  to 
identify  the  algorithm  parameters  and  to  extend  the  method  to  non-monotonic  approaches 
to  steady  state.  Kelton  and  Law’s  Monte  Carlo  results  are  used  as  a  base  line  for  comparison 
in  Chapter  VI. 

While  Kelton  and  Law’s  method  assumes  monotonic  convergence  to  a  steady-state 
mean,  Odoni  and  Roth  [73]  determine  initial  conditions  under  which  the  expected  number 
in  the  queue  does  not  move  monotonically  toward  the  steady-state  value.  Kelton  [42]  and 
Murray  and  Kelton  [71]  examine  the  transient  behavior  of  a  large  class  of  queuing  models 
and  also  note  non-monotonic  approach  to  the  steady-state  means.  They  also  demonstrate 
that  initial  conditions  greatly  affected  the  time  to  decay  within  a  specified  interval  of  the 
steady- state  value. 

The  same  year  Kelton  and  Law  published  their  techniques,  1983,  Welch  [101:290-294] 
proposed  a  truncation  procedure  that  is  relatively  easy  to  implement.  This  method  involves 
graphically  observing  the  convergence  of  the  mean  estimator.  Since  for  n  >  n<„  E[t/„]  = 
Hn  »  Welch  recommends  plots  of  the  output  sequence  {i/„}.  To  reduce  the  variance,  he 
recommends  averaging  the  t/n  across  replication  runs.  The  larger  the  number  of  replications, 
the  easier  the  stabilization  can  be  detected.  The  decision  can  be  simplified  further  if  the 
high  frequency  fluctuations  are  removed  with  a  moving  average.  The  moving  average  win¬ 
dow  size  is  increased  until  the  stabilization  on  the  plot  is  obvious  to  the  practitioner.  In 
addition,  Welch  suggests  plotting  confidence  intervals  about  the  means  for  various  n  to  give 
an  additional  indication  about  when  the  transient  effects  are  insignificant. 

Law  and  Kelton  [53:552]  state  that,  for  Welch’s  procedure  to  be  statistically  correct. 
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separate  sets  of  runs  should  be  used  to  identify  the  transient  phase  and  for  parameter  esti¬ 
mation.  They  state  that,  if  the  transient  phase  is  short  compared  to  the  run  length,  then  it 
is  “probably  safe”  to  use  the  same  set  of  runs  to  determine  the  warm-up  period  and  calculate 
parameter  estimates. 

Law  and  Kelton  [53:545-550]  further  recommend  Welch’s  [101:290-294]  plotting  tech¬ 
nique  with  subjective  assessment  as  the  “simplest  and  most  general”  approach  to  detect  the 
completion  of  the  trajisient  phase.  However,  this  technique  is  only  appropriate  for  univariate 
output  sequences.  Law  and  Kelton  also  state  that,  in  general,  one  replication  is  not  sufficient 
to  estimate  tIq. 

To  this  point,  this  literature  review  has  concentrated  on  identifying  the  effective  com¬ 
pletion  of  transient  phase  for  the  univariate  output  case.  Schruben  [86]  proposes  a  method 
to  select  a  truncation  point  for  multivariate  output  with  multiple  replications. 

For  a  multivariate  output  sequence,  initialization  bias  may  affect  some  output  measures 
and  be  insignificant  for  others.  Besides  serial  correlation  within  one  of  the  output  sequences, 
cross  and  lagged-cross  correlations  should  be  considered.  Schruben  [86]  demonstrates  the 
limitation  of  trying  to  examine  each  output  sequence  independently  for  stabilization.  He 
illustrates  a  case  in  which  each  of  the  output  sequences  has  stabilized,  but  the  correlation 
between  the  sequences  has  not  stabilized. 

Schruben’s  test  is  based  on  Hotelling’s  two  sample  statistic  [28:166-169].  Each 
output  vector  y„  consists  of  S  responses,  and  M  replications  are  run.  Schruben  notes  that 
to  use  Hotelling’s  requires  M  >  S,  and  he  further  recommends  that  M  —  S  >  5.  The 
procedure  begins  by  dividing  the  output  vectors  into  Nb  small  batches  with  batch  size  B  of 
5  to  10.  These  batches  are  not  constructed  to  reduce  serial  correlation  but  to  support  the 
normality  assumption  of  the  test  statistic.  An  average  vector  is  calculated  within  a  batch 
for  each  replication,  and  these  are  averaged  across  the  replications. 

1  ^ 

y.(WB)  =  =  1.2 . (14) 
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Presuming  that  the  run  is  long  enough  for  the  last  batch,  i  =  Nb,  is  at  equilibrium,  the 
sample  covariance  matrix  is  calculated  by 

1  M 

-  yN.(MB)f  (15) 

fFl=l 

When  the  number  of  replications  is  low,  Schruben  recommends  using  a  group  of  the  last 
batches  to  calculate  the  sample  covariance  matrix.  The  Hotelling’s  two-sample  test  statistic 
for  each  batch  i  is  given  by 

r;  =  -  y„.(MB)]’'S(Nfl)-My,(MB)  -  yH,(MB)\  (16) 

The  truncation  point  can  be  selected  when  shows  consistency  with  an  Fa^s,M-s  dis¬ 
tribution  [86].  Specifically,  beginning  with  the  first  batch,  batches  are  discarded  until 
Tft  <  Fa,s,M-s  for  3,  selected  value  of  a.. 

Schruben  evaluates  his  technique  through  a  Monte  Carlo  analysis.  He  graphs  the 
coverage  rates  based  on  the  desired  confidence  versus  the  observed  confidence  level  for  two 
sets  of  data.  The  first  data  set  is  the  remaining  observations  after  using  his  criteria,  and  the 
second  set  has  the  previous  “biased”  batch  added  in.  With  a  large  number  of  replications,  the 
effects  of  bias  becomes  much  more  apparent  because,  as  the  number  of  replications  increases, 
the  confidence  region  decreases.  For  a  biased  estimator,  the  confidence  region  shrinks  around 
the  incorrect  point.  Bias  is  therefore  apparent  by  the  observed  coverage  rate  of  the  confidence 
region  on  the  true  paxameter  being  considerably  lower  than  the  nominal  confidence  level. 
Schruben  tests  his  method  on  response  times  and  utilization  from  a  time-sharing  computer 
model  [3],  employing  100  replications  with  a  batch  size  of  3.  Using  his  truncation  criterion 
results  in  the  desired  confidence  level  equaling  the  observed  coverage  rate.  However,  when 
one  batch  before  the  appropriate  truncation  point  is  added  to  the  data  set,  the  observed 
confidence  level  decreases  significantly.  The  effectiveness  of  truncation  points  selected  by 
the  technique  proposed  in  Chapter  VII  are  compared  with  the  effectiveness  of  the  truncation 
points  selected  by  Schruben’s  algorithm. 
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2.4-S  Discrete-Event  Simulation  of  an  M/M /I  Queue.  The  M/M/1  queue  is  selected 
because  its  theoretical  mean  is  known  and  the  simulated  output  is  complex  and  highly 
variable.  Using  the  notation  that  the  mean  of  the  exponentially-distributed  interarrival 
times  is  j  and  the  mean  of  exponentially-distributed  service  times  is  the  traffic  intensity 
or  server  utilization  is  p  —  ^  (unitless). 

In  the  literature,  two  example  output  sequences  commonly  are  considered.  One  output 
is  waiting  time  in  the  system  for  a  sequence  of  entities.  This  is  an  example  of  output  that 
Pritsker  [77:36]  classifies  as  “statistics  based  on  observations”.  The  other  sequence  is  a 
periodic  output  of  the  number  in  the  simulation  queue,  which  is  an  example  of  “statistics 
based  on  time-persistent  variables”. 

The  expected  waiting  time  E[H/,]  in  steady  state  is 


EIW,)  = 


P 

fi  —  X 


with  variance 


Ap 

0^ 


The  steady-state  expected  length  of  the  queue  Efjh,]  is  given  by 


As  traffic  intensity  p  increases,  so  does  the  correlation  in  the  output.  Therefore,  as 
traffic  intensity  increases,  the  significant  effects  of  the  initial  transient  should  persist  longer. 
The  queue  becomes  unstable  and  grows  without  bound  when  p  >  1  since  entities  are  arriving 
faster  than  entities  are  served  on  the  average.  Figures  2  through  4  depict  simulations  where 
p  is  0.95  and  the  initial  conditions  are  an  empty  queue  and  an  idle  server  to  show  a  case 
with  high  autocorrelation  and  a  long  initial  transient. 

Figure  2  depicts  a  typical  sequence  of  waiting  times  for  entities  processing  through  an 
M/M/1  queue  with  first-in-first-out  service  order,  no  balks,  and  infinite  queue  capacity.  For 
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Figure  2.  Sample  Sequence  of  MjMfl  Queue  Waiting  Times  with  p  =  0.95  (E[Wg]  =  19) 
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this  example,  the  traffic  intensity  p  is  0.95  and  the  steady-state  expected  waiting  time  E[W,] 
is  19.0  time  units.  While  the  ratio  of  the  arrival  and  service  rates  determines  the  traffic 
intensity,  their  magnitudes  determine  the  average  waiting  time.  The  congestion  and  long 
waits  experienced  by  the  7,500th  through  9,500th  entities  are  typical  and  occur  periodically 
when  the  system  becomes  congested.  The  length  and  duration  of  long  lines  increase  with 
traffic  intensity. 

The  initial  condition  in  this  queue  is  empty  and  idle,  so  the  first  entity  through  has  no 
waiting  time,  Wq{to)  =  0.  Because  of  this  initial  condition,  the  probability  of  early  entities 
having  long  waiting  times  is  significantly  less  than  from  the  steady-state  output  distribution 
for  waiting  times.  For  the  system  to  be  stable,  that  is  p  <  1,  the  mean  arrival  rate  must  be 
less  than  the  mean  service  rate,  or  equivalently,  the  mean  interarrival  time  must  be  greater 
than  the  average  service  time.  Thus,  in  steady-state,  the  server  will  occasionally  be  idle,  and 
waiting  times  of  zero  occur.  The  empty  and  idle  initial  conditions  cause  the  transient  values 
to  have  lower  values,  but  not  necessarily  to  be  statistical  outliers  of  the  steady-state  output 
distribution.  The  initial  transient  values  simply  have  distributions  with  a  sequence  of  lower 
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Figure  3.  Average  of  30  Replications  of  M/M/1  Queue  Waiting  Times  (p  =  0.95) 
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means. 

Averaging  over  independent  replications  of  the  simulation  gives  an  empirical  indica¬ 
tion  of  the  time-dependent  output  distribution  mean.  Figure  3  shows  the  average  of  30 
output  sequences.  The  first  observation  in  each  sequence  is  zero,  and  by  about  the  1000th 
observation,  the  average  is  very  near  the  steady-state  average  of  19. 
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Figure  4.  Welch’s  Technique  with  Moving  Average  (Window  =  500) 


Welch’s  technique  [101],  which  is  described  earlier,  recommends  processing  the  sequence 
of  averages  through  a  moving  average.  In  the  moving  average,  each  observation  is  replaced 
with  the  average  of  a  set  number  of  preceding  observations,  the  same  number  of  following 
observations,  and  the  original  observation.  The  number  of  proceeding  oberservations  is  called 
the  moving  average  window  size.  At  the  beginning  and  end  of  the  sequence,  the  average  is 
taken  over  the  points  that  would  be  in  the  complete  window.  Welch’s  technique  assumes 
that  the  steady-state  distribution  is  gradually  approached  so  that  the  output  distribution 
of  nearby  observations  should  be  very  similar.  The  moving  average  smooths  the  sequence 
of  averages  across  replications.  Welch  recommends  increasing  the  window  size  until  the 
transient  is  apparent.  Figure  4  depicts  the  moving  average  with  a  window  size  of  500  for  the 
average  of  30  output  sequences.  Since  the  sequence  appears  to  become  level  for  the  1000th 
to  4000th  observation,  the  initial  transient  appears  effectively  to  be  complete  by  the  1000th 
observation. 
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2.4-4  Requirements  for  a  Good  Truncation  Point  Selection  Method.  Gafarian,  Ancker 
and  Morisaku  [24]  use  five  criteria  for  comparing  truncation  techniques.  Their  criteria  are 
accuracy,  precision,  generality,  cost,  and  simplicity.  The  first  two,  accuracy  and  precision, 
relate  only  to  situations  with  predefined  “true”  truncation  points.  Confidence  interval  half 
widths  and  coverage  rates  are  generally  accepted  as  the  best  criteria  for  measuring  the 
success  of  a  technique.  Generality  means  that  “the  rule  performs  well  across  a  broad  range 
of  systems  and  a  broad  range  of  parameters  within  a  system.”  Cost  reflects  the  computer 
time  required  for  the  algorithm.  However,  since  simulation  models  themselves  can  require 
significant  amounts  of  computer  time,  almost  any  technique  will  have  negligible  impact  on 
the  total  computer  time  required.  Simplicity  ensures  that  the  rule  is  usable  by  the  average 
simulation  practitioner  and  implies  that  the  technique  should  not  require  too  many  variables 
to  be  set  by  the  user.  Past  techniques  axe  criticized  for  being  too  sensitive  to  parameter 
settings  and  for  not  providing  rules  to  specify  the  necessary  parameter  settings  [105,  77:753]. 

General  recommendations  are  against  trying  to  determine  steady  state  based  on  cumu¬ 
lative  statistics.  Their  values  typically  lag  behind  the  current  statistics,  resulting  in  excessive 
deletion  of  data  [15]. 

2. 5  Summary 

The  typical  assumptions  and  issues  for  steady-state  output  analysis  of  discrete-event 
simulations  are  discussed.  Common  assumptions  embedded  in  most  analysis  techniques  are 
covariance  stationarity  and  ergodicity  of  the  output  sequence.  Covariance  stationarity  in  a 
univariate  sequence  implies  that  the  output  sequence  has  a  constant  mean,  finite  and  constant 
variance,  and  the  covariance  for  any  two  observations  separated  by  any  set  interval  is  the 
same.  For  discrete-event  simulation  output  analysis,  ergodicity  implies  that  the  estimator  of 
the  mean  is  unbiased  if  its  variance  decreases  to  zero  as  the  number  of  observations  increases 
to  infinity. 

Output  analysis  issues  include  dealing  with  the  initial  transient  and  correlated  output. 
The  initial  transient  is  induced  since  the  initial  conditions  are  not  representative  of  unknown 
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steady-state  conditions.  The  typical  approach  of  dealing  with  the  transient  or  warm-up 
period  is  initial  data  deletion  or  truncation.  Deleting  too  much  initial  data  increases  the 
vaxiance  of  the  steady-state  parameter  estimates,  whereas  not  deleting  enough  data  may 
result  in  biased  estimates.  Determining  the  appropriate  truncation  point  is  the  objective  of 
this  research. 

Positive  correlation  in  simulation  output  reduces  the  information  obtained  from  each 
additional  observation.  Since  the  statistical  techniques  for  independent  observations  do  not 
apply  directly,  each  output  analysis  technique  must  deal  with  the  correlation  of  the  output. 

The  next  major  section  in  this  chapter  deals  with  steady-state  or  initial  transient 
identification.  In  other  words,  steady-state  identification  is  determining  the  point  where 
the  sequence  should  be  truncated  to  eliminate  the  estimation  bias  induced  by  the  initial 
conditions.  While  the  steady-state  output  distribution  is  only  reached  in  the  asymptotic 
limit  as  run  length  increases,  the  simulation  output  passes  a  point  where  steady  state  is 
effectively  attained.  This  means  that  the  error  induced  by  assuming  that  steady  state  has 
been  attained  is  statistically  insignificant. 

The  section  on  steady  state  identification  reviews  the  literature  for  evaluation  criteria. 
Currently,  the  most  widely  accepted  criteria  are  the  resulting  confidence  interval  width  and 
realized  coverage  rates  for  the  truncated  sequences.  Schruben’s  [87]  initialization  bias  test 
can  also  be  used. 

The  literature  review  continues  with  proposed  truncation  point  selection  algorithms 
and  published  evaluations  of  these  algorithms.  A  general  conclusion  of  the  literature  is  the 
lack  of  a  sound  analytical  approach  for  the  truncation  of  a  univariate  output  sequence.  The 
chapter  concludes  with  criteria  for  a  good  initial-data  truncation  method. 

One  of  the  major  sources  of  difficulty  in  simulation  output  analysis  is  a  result  of 
correlation  in  the  output  sequence.  In  both  Schruben’s  [87]  and  Kelton  and  Law’s  [44] 
approaches,  the  output  sequence  is  assumed  to  be  representable  as  a  constant  steady-state 
mean  plus  noise.  Since  the  output  sequence  is  generally  correlated,  the  mean  being  constant 
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implies  the  noise  accounts  for  the  correlation.  Schruben  [87]  assumes  the  noise  is  ^-mixing, 
but  he  only  demonstrates  that  the  transformation  to  a  Brownian  bridge  works  if  the  noise 
sequence  is  independent.  Kelton  and  Law  [44]  use  generalized  least-squares  to  account  for 
the  autocorrelation. 

In  a  different  approach  to  account  for  correlation  in  the  output  sequence,  this  research 
applies  a  filter  with  an  autoregressive  model  to  account  for  the  correlation  in  the  noise. 
Chapter  III  presents  the  linear  system  theory  and  Kalman  filter  equations  for  this  approach, 
and  univariate  truncation  point  selection  methodologies  based  on  these  concepts  are  proposed 
in  Chapters  IV  and  V.  Although  the  residual  monitoring  approach  in  Chapter  IV  is  not  as 
successful  as  the  MMAE  approach  in  Chapter  V,  Section  4.2  provides  the  model  formulation 
used  throughout  this  research.  The  transient  effects  are  less  significant  with  a  single  run, 
so  the  truncation  method  is  applied  to  cases  with  multiple  replications  in  Chapters  VI 
and  VII.  Chapter  VI  provides  the  algorithm  and  results  for  univariate  output  sequences, 
eind  in  Chapter  VII  the  method  is  extended  for  multivariate  output  sequences.  Chapter  VIII 
summarizes  this  research. 
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III.  Kalman  Filters  and  Model  Estimation 


S.  1  Introduction 

KalmaJi  [40]  and  KaJman  and  Bucy  [41]  developed  the  Kalman  filter.  Kalman’s  contri¬ 
bution  wa.s  to  combine  the  work  of  the  German  mathematician  Gauss  with  the  state-space 
vector  representation  from  linear  system  theory.  The  state  vector  is  a  set  of  variables  such 
that  the  present  state  vector  with  future  inputs  completely  describe  system  behavior.  This 
definition  of  state-space  representation  is  limited  to  Markov  processes  since  the  Markov 
property  implies  that  some  set  of  sufficient  statistics  at  the  current  time  provides  as  much 
information  about  the  system  as  the  complete  time  history  of  the  process.  In  a  state-space 
representation,  system  equations  are  written  to  depict  the  state  dynamics  and  measurement 
relationships.  Kidman’s  original  de.nvation  is  based  on  the  fact  that  the  updated  state  esti¬ 
mate  at  time  tn,  is  the  orthogonal  projection  of  the  true  state  x(t„)  onto  the  subspace 

spanned  by  the  measurement  history  Z(t„)  [57:235]. 

The  Kalman  filter  is  an  optimal  recursive  next-step  prediction  algorithm.  After  ini¬ 
tialization,  the  discrete  Kalman  filter  continues  with  a  two-step  procedure.  The  first  step 
propagates  the  best  estimate  of  the  state  through  time.  The  second  step  uses  the  information 
contained  in  a  noise- corrupted  measurement  to  update  the  prediction.  The  algorithm  repeats 
again  by  propagating  this  estimate  to  the  next  time  interval.  These  steps  of  propagation 
and  update  are  repeated  for  each  of  the  available  observations. 

The  notation  employed  in  this  chapter  follows  engineering  control  theory  conventions, 
specifically  Maybeck  [57,  58,  59].  Estimated  parameters  are  indicated  with  a  hat,  and  vectors 
are  in  boldface,  such  as  the  state  vector  estimate  x(tn)-  At  each  point  in  time,  two  state 
estimates  are  used;  the  propagated  estimate,  prior  to  the  measurement  update,  is  indicated 
with  a  minus,  x(i~),  and  the  state  estimate  after  the  measurement  update  is  marked  with 
a  plus,  x(t^).  Alternative  Kalman  filter  notation  and  descriptions  include  Meinhold  and 
Singpurwalla  [68]  and  Harvey  [33]. 
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Simulation  output  is  a  discrete  realization  of  a  stochastic  process.  In  the  next  section, 
the  stochastic  process  is  approximated  a.s  the  solution  to  a  linear  stochastic  differential  equa¬ 
tion  driven  by  Brownian  motion.  The  stochastic  process  is  defined  in  terms  of  the  states  that 
cire  estimated  by  the  Kalman  filter,  which  may  or  may  not  be  the  simulation  states  or  the  sim¬ 
ulation  observations.  Since  simulation  output  calculated  with  a  digital  computer  is  discrete, 
rather  than  using  the  differential  equation  directly,  an  equivalent  discrete-time  difference 
model  [57:170]  is  applied.  The  discrete-time  Kalman  filter  is  discussed  in  Section  3.3.  The 
final  major  section  in  this  chapter  is  on  system  identification.  While  techniques  for  system 
identification  are  reviewed  after  describing  the  Kalman  filter,  the  actual  model  formulation 
and  system  identification  are  not  discussed  until  the  next  chapter. 

S.  2  Stochastic  Processes 

Before  showing  the  Kalman  filter  formulas  in  the  next  section,  some  underlying  theory 
of  stochastic  processes  is  discussed.  This  section  presents  a  linear  stochastic  diferential 
equation  and  the  cissociated  dynamics  model  and  measurement  model. 

8.2.1  Linear  Stochastic  Differential  Equations.  Assume  the  underlying  stochastic 
process  can  be  described  or  approximated  with  a  linear  stochastic  differential  equation, 

dx(t)  =  F(t)x(f)dt  -1-  G{t)d(3{t)  (17) 

where  x  is  a  vector  of  the  filter-design  system  states 
F  is  the  dynamics  system  matrix 

G  determines  which  system  states  are  affected  by  noise 
13  is  the  dynamics  driving  noise 

The  system  is  linear  if  F(t)  is  not  a  function  of  the  state  values  x(t).  Further,  assume  that 
the  dynamics  driving  noise  can  be  modeled  as  Brownian  motion.  Brownian  motion  has  the 
following  characteristics  [57:148,155]: 
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•  Brownizm  motion  has  independent  increments  in  time,  and 

•  the  increments  are  normally  distributed  random  variables  such  that  for  any  time  in- 
steints,  t  and  t' , 

EtaCi))  =  0 

E((^(i)  -  /3(t'))(/3(t)  -  mf]  =  /.'• 

/3(to)  =  0  by  convention 

The  matrix  Q(t)  represents  the  diffusion  of  the  process.  The  term  diffusion  is  from  an 
early  application  of  Brownian  motion  for  modeling  the  movement  of  gas  molecules.  In  other 
applications,  Q(t)  determines  how  faist  the  states  in  the  process  are  changing. 

Heuristically,  Brownian  motion  is  the  result  of  white  (uncorrelated  in  time)  noise  passed 
through  an  integrator.  In  reverse,  the  hypothetical  derivative  of  Brownian  motion  is  white 
noise.  The  derivative  is  hypothetical  because  Brownian  motion  is  nondifferentiable.  Heuris¬ 
tically,  Brownian  motion  is  a  continuous  process  which  has  corners  everywhere  [57:152]. 

A  discrete  sample  of  scalar  white  noise  generated  by  sequencing  100  independent  stan¬ 
dard  normal  random  variables  is  shown  in  Figure  5.  The  notation  Wci{tn,u;i)  indicates  w  for 
white  noise,  subscript  d  for  discrete,  for  time,  and  tvi  to  indicate  which  realization  from 
the  sample  space  Q.  The  cumulative  sum  of  the  white  noise  of  Figure  5  is  a  discrete  sample 
sequence  of  Brownian  motion  with  unit  diffusion,  Q(0  =  1.  Three  realizations  are  depicted 
in  Figure  6.  Similarly,  0  indicates  Brownian  motion,  is  for  time,  and  w,  indicates  the 
realization  from  the  sample  space.  If  the  diffusion  strength  Q(^)  is  increased,  the  variance 
in  the  reeilizations  of  the  Brownian  motions  also  increases.  In  other  words,  the  variance  or 
mean  squared  value  of  the  process  grows  at  a  greater  rate.  The  vertical  scales  in  the  figures 
can  be  adjusted  to  depict  other  than  Brownian  motion  with  unit  diffusion  strength. 
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S.2.2  Dynamics  Equation.  The  general  time- varying  solution  to  the  linear  stochastic 
differential  equation  (17)  can  be  shown  to  be  [57:40,164] 

x(t)  =  $(i,io)x(to)  +  /  ^(t,T)G(T)d/3(T) 

Jto 

where  $(t_to)  is  the  state  transition  matrix  from  time  to  to  t  that  satisfies  the  differential 
equation  and  initial  conditions: 


d[$(t,to)]/dt  =  F(t)$(t,  to) 


^(to,  to)  —  I 

When  the  dynamics  system  matrix  F(t)  is  a  constant  matrix,  the  system  is  described  by  a 
linear  time-invariajit  differential  equation,  and  the  transition  matrix  can  be  calculated  using 
^(t,  to)  =  ^(t  -  to)  =  or  in  the  Laplace  domain  $(s)  =  [si  —  F]”^  [57:41],  Since  this 

research  is  limited  to  simulation  models  with  output  that  attains  a  stationary  steady-state 
distribution,  only  time-invariant  system  models  are  used. 

Since  the  simulation  output  is  a  set  of  discrete  realizations  or  observations  of  the 
stochastic  process,  for  this  application  an  equivalent  discrete-time  model  is  useful.  Beginning 
with  the  continuous- time  solution  and  integrating  over  time  results  in  the  discrete-time 
version  of  the  dynamics  equation. 


x(t„+i)  =  $(t„+i,t„)x(t„)  +  Wj(t„) 
where  wj(i„)=  f  ^{t„+i,T)G{T)d0{T) 

The  transition  matrix  $  relates  the  state  vector  x(t„)  at  one  time  to  the  state  vector  at 
the  next  time  index  x(t„+i).  Since  the  continuous-time  dynamics  driving  noise  is  assumed 
to  be  Brownian  motion,  the  integration  or  summation  of  normally-distributed  Brownian 
increments  is  also  normally  distributed.  Therefore,  discrete-time  dynamics  driving  noise 
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is  normally  distributed  with  zero-mean  and  the  following  characteristics: 

E[w<i(t„)]  =  0 

E[w<f(t„)wJ(t„)]  =  $(t„+i,r)G(T)Q(T)G^(T)$^(t„+i,T)dT  =  Qd(t„)  (18) 

E[Wrf(ti)wJ(tj)]  =  0  for  t,  ^  tj 

The  third  characteristic  is  a  result  of  the  Brownian  motion  having  independent  increments, 
which  makes  the  dynamics  noise  sequence  {wd(tn)}  uncorrelated  in  time  or  white. 

While  discrete-time  dynamics  equations  are  often  derived  from  continuous-time  rela¬ 
tionships,  discrete-time  dynamics  which  do  not  have  corresponding  continous-time  equations 
are  also  useful. 

x(t„+i)  =  -f  Gd(t„)wj(t„)  (19) 

For  example.  Equation  (19)  incorporates  a  G<i(t„)  that  determines  which  filter-design  states 
the  noise  affects.  This  process  only  has  a  corresponding  continuous  process  if  the  resulting 
dynamics  noise  covariance  matrix  Qd(tn)  is  of  full  rank.  In  this  research,  the  formulation 
in  Equation  (19)  is  used.  Since  only  discrete  output  sequences  are  considered,  the  lack  of  a 
corresponding  continuous  relationship  has  no  adverse  impact. 

Assuming  equally-spaced  samples  and  a  time-invariant  system  model,  the  transition 
matrix  is  constant  t„)  =  For  application  to  simulation  output,  equal  spacing  can 

be  achieved  in  terms  of  simulated  time  or  an  index  for  the  output  sequence.  For  example, 
customers  processing  through  a  queuing  system  can  be  numbered  sequentially  for  an  index 
[20].  With  the  time  index  formulation,  the  transition  matrix  represents  the  relationship 
between  subsequent  entities.  In  addition,  since  steady-state  simulation  output  is  modeled, 
the  dynamics  noise  Wj(t„)  is  assumed  to  have  constant  variance,  Q<i(tn)  =  Qd,  and  the 
distribution  matrix  G(t)  is  also  assumed  to  be  a  constant  G  throughout  the  simulation 
process. 
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S.2.S  Measurement  Model.  In  order  to  apply  the  Kalman  filter,  measurements  of  the 
system  states  are  necessary.  In  these  applications,  the  simulation  output  is  used  to  determine 
discrete  measu /ements  z„  of  the  filter-design  states  x(t„).  The  measurement  model  indicates 
the  relationship  between  the  system  states  and  the  measurements: 

z(t„)  =  H(<„)x(t„)  -f  v(i„)  (20) 

where  z  is  the  measurement  or  observation 

H  is  the  measurement  or  observation  matrix 
X  is  the  vector  of  filter-design  system  states 
V  is  the  measurement  noise 

Assume  that  the  measurement  noise  sequence  {v(i„)}  is  white  (uncorrelated  in  time), 
normal!  ■'istributed,  zero-mean  stochastic  sequence,  such  that, 

B(v(i„)l  =  0 

E[v((„)v’'{i„)|  =  R((„)  (21) 

£[v((i)vl'(f,)]  =  0  for  u  ^  tj 

If  the  measurement  matrix  H(t„)  is  not  a  function  of  the  states  x(t),  then  the  mea¬ 
surement  equation  is  linear.  In  addition,  for  steady-state  simulation  output  sequences,  the 
measurement  matrix  H(t„)  and  the  variance  of  the  measurement  noise  process  R(t„)  are  as¬ 
sumed  not  to  change  with  time.  Therefore,  these  time-dependent  matrices  are  replaced  with 
constant  matrices,  H(^„)  =  H  and  R(t„)  =  R.  In  addition,  the  noise  sequences  {wj(i„)} 
and  {v(t„)}  generally  are  modeled  to  be  uncorrelated  with  each  other. 

This  section  reviews  stochastic  processes.  For  a  linear  stochastic  differential  equation, 
the  dynamics  model  and  measurement  model  are  shown.  Based  on  these  equations,  the  next 
section  describes  the  Kalman  filter. 
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S.S  Linear  Gaussian  Time- Invariant  Kalman  Filter  Algorithm 

The  discrete-time  Kalman  filter  algorithm  is  shown  for  a  time-invariant,  linear  system, 
with  no  control  inputs,  and  normally-distributed  (Gaussian)  zero-mean  discrete  dynamics 
noise  and  measurement  noise  [57:275].  “Discrete-time”  implies  that  the  propagation  and 
measurements  updates  occur  only  at  set  intervals,  in  contrast  to  many  continuous-time 
engineering  applications.  “Linear”  implies  that  the  values  of  the  filter-design  system  states 
x(<„)  do  not  affect  the  values  of  the  transition  matrix  the  noise  input  matrix  G,  or  the 
measurement  matrix  H.  “Time-invariant”  means  that  the  matrices  ^  ,  G,  and  H  do  not 
change  throughout  the  stochastic  process.  The  covariance  matrices  of  the  noises  Qj  and  R 
are  constant  since  the  process  variance  is  assumed  stationary. 

In  this  section,  the  two  stages  of  the  Kalman  filter,  the  propagation  stage  and  the 
measurement  update  stage,  are  shown.  An  example  of  a  Kalman  filter  is  depicted.  The 
section  also  includes  discussions  on  constant  gain  Kalman  filters  and  optimality  of  the  filter’s 
state  estimates. 

3.3.1  Propagation  Stage.  Two  equations  comprise  the  propagation  stage.  The  first 
relationship  determines  the  propagation  through  time  of  the  estimated  state  vector.  The 
second  equation  propagates  the  covariance  matrix  of  the  state  variables  through  time.  The 
propagation  equation  takes  the  best  state  estimate  at  the  previous  time  (or  the 

initial  estimate  x(to))  and  moves  it  through  time  by  multiplying  by  the  transition  matrix 
Therefore,  the  propagation  equation  is 

x{t-)  =  $x(t+_i)  (22) 

where  x  is  the  estimated  state  vector 
$  is  the  transition  matrix 

This  estimate  results  from  computing  the  conditional  expectation  of  the  dynamics  model. 
Equation  (19),  since  the  dynamics  driving  noise  Wj(t„)  has  a  mean  of  zero. 
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The  associated  covariance  matrix  P(i„)  of  the  state  estimate  x(t„)  is  calculated  with 

P(i;)  =  +  GjQjGj  (23) 

where  P  is  the  covariance  matrix  of  state  estimates  x 
Gd  is  the  dynamics  noise  wj(tn)  input  matrix 
Qrf  is  the  covariance  matrix  of  the  discrete  dynamics  driving  noise 

These  two  equations,  one  for  the  state  estimate  and  the  other  for  the  associated  co- 
variance,  complete  the  propagation  stage.  Each  propagation  is  followed  by  a  measurement 
update,  and  then  the  two-stage  cycle  begins  again. 

3.S.S  Measurement  Update  Stage.  After  the  propagation  stage,  the  second  step  is  the 
measurement  update.  The  measurement  update  is  actually  a  static  estimation  problem  of 
combining  two  separate  sets  of  information.  A  state  estimate  results  from  the  dynamics  equa¬ 
tion  in  the  propagation  stage.  The  correction  to  that  state  estimate  is  based  upon  the  actual 
measurement  and  the  measurement  model.  Both  estimates  are  normally  distributed  with 
known  covariance  matrices.  The  Kalman  filter  gain  K(tn)  provides  the  weighting  between 
the  two  sets  of  information  about  the  state. 

K(t„)  =  P(i;)H’'[HP(i;)H’'  +  RJ-'  (24) 

where  H  is  the  measurement  or  observation  matrix 
R  is  the  measurement  covariance  noise  matrix 
K  is  the  Kalman  gain 

The  measurement  update  equations,  shown  below,  determine  the  new  state  estimates 
and  covariance  matrix  after  the  measurement  at  time  tn  is  incorporated.  The  new  updated 
state  estimate  is 

x(C )  =  )  +  K(t„)[zn  -  Hx(t; )]  (25) 

where  is  the  actual  measurement  at  time 
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The  updated  state  estimate  x(tj^)  is  the  previous  state  estimate  x(t“)  corrected  with  the 
Kalman  filter  gain  times  the  new  information  provided  by  the  measurement.  The  new  infor¬ 
mation  is  obtained  from  the  measurement  by  taking  the  actual  measurement  2„  and  subtract¬ 
ing  the  best  prediction  of  the  measurement  before  the  measurement  is  received.  Hx(t")  is 
the  prediction  of  the  measurement  based  on  the  assumed  measurement  model,  Equation  (20), 
and  the  fact  that  the  measurement  noise  v(<„)  has  a  mean  of  zero.  The  differences  between 
the  actual  measurements  and  the  predicted  measurements  are  the  residuals. 

The  residual, 

r(t„)  =  z.-Hx(t;)  (26) 

can  be  viewed  as  the  new  information  contained  in  the  measurement.  Under  the  assumptions 
of  line?’-,  Gaussian  (normally-distributed  noise  terms),  time-invariant  models  with  known 
system  matrices  ($,  G^,  Qj,  H,  and  R),  the  residuals  are  normally  distributed  with  a  mean 
vector  of  E[r(in)]  =  0  and  covariance  matrix  of  E[r(i„)r^(t„)]  =  HP(t~)H^-FR.  In  addition, 
the  residual  sequence  {r(in)}  is  a  white  (uncorrelated  in  time)  sequence  [57:228-229].  The 
new  information  contained  in  the  residual  r(t„)  may  be  substituted  into  Equation  (25)  to 
get  a  simplier  form  of  the  state  updated  equation. 

x(C)  =  x(t;)  +  K(t„)r(t„)  (27) 

The  Kalman  filter  residuals,  from  Equation  (26),  are  critical  in  the  simulation  applica¬ 
tions  of  the  Kalman  filter.  Besides  representing  the  new  information  in  each  successive  mea¬ 
surement,  the  residuals  can  be  thought  of  as  the  error  in  the  dynamics  equation  prediction  as 
projected  onto  the  measurement  space.  Thus,  one  approach  to  estimating  unknown  parame¬ 
ters  in  the  dynamics  and  measurement  equations  is  selecting  parameters  which  minimize  the 
magnitude  of  the  residuals.  The  common  technique  is  to  use  least  squares  estimation,  which 
minimizes  the  sum  of  the  squared  residuals.  If  the  steady-state  model  of  the  simulation  out¬ 
put  can  be  approximated  with  an  appropriate  linear  model,  the  residuals  should  constitute 
a  white,  normally  distributed,  zero  mean  sequence.  The  Kalman  filter  residuals  may  provide 
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an  indication  of  the  adequacy  of  the  estimated  model’s  fit  to  the  data  being  processed. 

Along  with  an  updated  state  estimate,  an  updated  covariance  matrix  is  also  necessary. 
Since  the  Kalman  filter  gain  K(t„)  is  the  weight  based  on  comparing  the  variances  of  the 
estimate  from  the  propagation  stage  and  the  measurement  model,  it  stands  to  reason  that 
this  same  gain  is  used  to  determine  the  reduction  in  variance  resulting  from  incorporating 
the  measurement  information: 


P(C)  =  P(i;)-K(i„)HP(t;)  (28) 

In  the  scalar  case,  since  at  worst  the  measurement  contributes  no  new  information,  the 
updated  variance  is  always  less  than  or  equal  to  the  propagated  variance,  P(f^)  <  P{K)- 
Then  during  the  propagation  phase,  the  variance  increases  to  P{tn+i )  because  of  the  uncer¬ 
tainty  induced  by  the  dynamics  driving  noise. 

The  Kalman  filter  gain  K(tn)  weights  the  information  provided  by  the  dynamics  equa¬ 
tion  and  the  measurement  information  according  to  their  variances.  As  the  terms  of  Qj,  and 
hence  K(t„),  increase,  the  measurement  information  is  weighted  heavier  and  the  dynamics 
model  prediction  is  weighted  less.  Conversely,  cis  the  terms  of  R  increase,  and  thus  elements 
of  K(t„)  decrease,  the  measurement  information  is  weighted  less. 

For  a  moment,  consider  a  scalar  state  vector  i(tn)-  If  measurements  of  this  state  are 
available,  the  measurement  matrix  H  —  1.  Under  these  conditions,  the  Kalman  filter  gain 
simplifies  to  K{tn)  =  If  the  measurements  are  perfect,  the  measurement  noise 

variance  is  zero,  R  =  0.  With  iZ  =  0,  the  Kalmcin  filter  gain  is  one,  K  =  1,  and  thus 

=  2„.  Therefore,  the  update  equation  completely  disregards  the  prediction  based  on 
the  dynamics  equation  of  the  propagation  stage,  and  the  updated  estimate  is  based  entirely 
on  the  measurement  model  prediction.  This  is  logical  since  the  measurements  of  the  state 
are  perfect. 

At  the  opposite  end  of  the  spectrum,  if  the  measurement  is  so  poor  that  it  provides 
no  additional  information,  the  measurement  noise  variance  equals  infinity,  R  =  oo.  Under 
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these  conditions,  the  Keilman  filter  gain  is  zero,  X  =  0.  Thus,  the  update  step  totally 
disregards  the  worthless  measurement,  and  the  updated  prediction  would  be  based  entirely 
on  the  propagation  equations,  x{t^)  =  x(t~). 

In  actual  practice,  both  the  propagation  state  estimate  and  the  measurement  model 
state  estimate  are  combined  for  the  best  estimate.  The  weights  applied  are  determined  by 
the  Kalman  filter  gain  which  accounts  for  the  different  quality  of  their  information  by  means 
of  their  variances. 

Another  important  observation  is  that  the  covariances  P(t^)  and  do  not  depend 

on  the  actual  measurements.  Inspection  of  Equations  (23),  (24)  and  (28)  shows  the  covariance 
matrices  P(t^)  and  P(<^)  are  unaffected  by  the  actual  measurements  z„.  In  fact,  with 
constant  matrices,  ^,Gj,  Qj,  H  and  R,  the  covariance  matrices,  P(<~)  and  P(<ji^),  and 
hence  the  Kalman  filter  gain  vector  K(t„),  attain  constant  matrices,  P~,P'‘',  and  K,  as  the 
influence  of  initial  values  of  P(to)  decays.  Therefore,  these  values  can  be  precomputed  and 
stored  prior  to  the  actual  running  of  the  Kalman  filter. 

Besides  the  variance  estimates  being  independent  of  the  actual  measurements,  the 
magnitude  of  the  constant  gain  K  depends  only  on  the  relationship  between  Qj  and  R,  not 
their  actual  values.  In  the  case  of  scalar  noises,  the  ratio  ^  determines  the  gain  K.  This 
indeterminacy  complicates  simultaneous  estimation  of  both  Qj  and  R. 

3.S.S  Kalman  Filter  One  Dimensional  Example.  In  order  to  illustrate  the  use  of  the 
Kalman  filter,  a  simple  scalar  example  is  shown.  The  initial  estimate,  variance,  and  the 
system  parameters  are  as  follows: 

Initial  state  estimate  xq  =  100 
Transition  matrix  $  =  0.9 
Measurement  matrix  H  —  \ 

Dynamic  noise  input  matrix  Gd  =  I 

The  initial  state  estimate  is  normally  distributed  with  mean  of  Iq  and  variance  of  Pq. 


Initial  variance  Fq  =  1 
Dynamics  noise  variance  Qd  —  5.2 
Measurement  noise  variance  R  =  3.0 
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Kalman  Filter  State 

Figure  7.  Propagation  Through  Time 


The  propagation  stage  moves  the  normal  probability  density  of  the  initial  state  estimate 
forward  ia  time  to  ti: 

x(<r)  =  $x(to)  =  0.9(100)  =  90 

P(fr)  =  +  GaQjGj  =  (0.9)2(1.0)  +  (1)5.2(1)  «  6.0 


The  propagation  stage  maintains  the  normal  probability  density  with  a  mean  of  x{t~)  and  a 
variance  of  P{t~).  Figure  7  depicts  the  move  of  the  normal  distributed  state  estimates  from 
a  mean  100  at  time  to  to  a  mean  of  90  at  time  tj  with  an  associated  increase  in  variance 
due  to  th  *  dynamics  driving  noise. 

The  measurement  update  at  ti  incorporates  the  information  contained  in  a  measure¬ 
ment  received  at  that  time.  For  this  example,  let  the  measurement  be  Zi  =  84: 


Kit,) 

x{tt) 

Pitt) 


l(6.0)l+3 


=  i(tj  )  -f  K{t,)[z,  -  Hxity)]  =  90  +  ?[84  -  1(90)]  =  86 
=  P(tr)  -  K{t,)HPit:[)  =  6.0  -  f(l)6.0  =  2.0 
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Figure  8.  Measurement  Update 

From  the  measurement  model,  z{tn)  =  Hx(t„)  +  v(in)  with  If  =  1  and  an  actual  measure¬ 
ment  Zn,  an  estimate  of  the  state  conditioned  only  on  the  new  measurement  equals 

Zn  —  v(tn)-  Although  the  measurement  noise  v(tn)  is  unknown,  it  is  normally  distributed 
with  a  zero  mean  and  a  variance  of  R{tn).  Therefore,  the  state  estimate  conditioned  on 
the  new  measurement  at  time  t„,  is  also  normally  distributed  with  a  mean  value 

corresponding  to  the  measurement  and  the  variance  of  the  measurement  noise  R{tn). 

The  Kalman  filter  gain  K{tn)  provides  the  appropriate  weight  to  combine  these  two 
normal  probability  distributions.  The  propagated  distribution  with  a  mean  of  x{t~ )  receives 
a  weight  of  1  —  K(tn),  and  the  distribution  conditioned  on  the  new  measurement  with  mean  of 
i\zn{tn)  is  weighted  by  K(t„).  Since  this  is  a  linear  combination  of  two  normally-distributed 
random  variables,  the  resulting  estimate  x(<^)  is  also  normally  distributed.  Furthermore, 
since  K{tn)  is  bounded  between  zero  and  one,  the  resulting  distribution  has  a  mean  between 
the  previous  two  means.  Figure  8  shows  the  distributions  of  the  propagated  state  estimate 
x{t^ ),  the  state  estimate  conditioned  on  the  new  measurement  x\zi{ti),  and  the  updated  state 
estimate  x{ti  ).  The  updated  state  estimate  has  smaller  variance  than  either  the  propagated 
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state  estimate  or  the  state  estimate  conditioned  on  the  measurement. 

The  recursive  nature  of  the  Kalman  filter  algorithm  is  that  it  repeats  the  steps  again 
beginning  with  the  updated  state  estimates.  The  two  steps  of  propagation  and  update  are 
repeated  for  each  observation. 

S.S.4  Constant  Kalman  Filter  Gain.  As  mentioned  previously,  the  Kalman  filter  gain 
K(<„)  eind  the  covariance  matrices  P{t~)  and  P(t^)  are  not  dependent  upon  the  actual  mea¬ 
surements.  Actually,  these  matrices  are  functions  of  the  initial  estimate  P(to))  fhe  covariance 
of  the  dynamics  driving  noise  Q<i(t„),  and  the  measurement  noise  covariance  R(<n).  Since 
Qj  and  R  are  constant  (as  well  as  $,Gj,  and  H)  in  these  applications,  K(i„),P(t"),  and 
P(t^)  attain  steady-state  vaJues  as  the  contribution  of  P(to)  decays.  These  steady-state  val¬ 
ues  can  be  determined  from  the  Kalman  filter  equations  by  setting  the  the  variance  matrices 
to  constant  matrices.  The  constant  variance  version  of  Equation  (23)  is 

p-  =  -I-  GdQdGj  (29) 

From  Equation  (24),  the  steady-state  Kalman  filter  gain  is 

K  =  P-H^[HP-H^  +  RJ-'  (30) 

and  from  Equation  (28), 

P+  =  P--KHP-  (31) 

Substituting  for  P"^  and  K  into  Equation  (29)  results  in 

p-  _  +  ^P-H^[HP-H^  +  R]-'HP-$^  -  GdQjGj  =  0  (32) 

This  equation  is  called  an  algebraic  Riccati  equation  because  of  its  form.  For  the  previous 
one-dimensional  example,  the  progression  of  these  values  is  in  Table  5  for  calculations  to  two 
decimal  places.  The  state  estimate  variances  and  the  Kalman  filter  gain  rapidly  converge  to 
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Table  5.  Steady-State  Variances  and  Gain 


in 

P{in) 

P{tt) 

0 

1.00 

1 

6.01 

0.67 

1.98 

2 

6.81 

0.69 

2.11 

3 

6.91 

0.70 

2.07 

4 

6.88 

0.70 

2.06 

5 

6.87 

0.70 

2.06 

their  steady-state  values. 

Using  the  steady-state  matrices,  P~,  P"*",  K,  the  two-step  Kalman  filter  algorithm 
simplifies  to  only  two  equations,  Equations  (22)  and  (25)  with  a  constant  Kalman  gain  K. 
The  covariance  matrices  are  not  cadculated  since  their  steady-state  values  are  used  for  all 
time.  For  the  simple  example  shown  previously,  the  steady-state  values  are  used  to  generate 
the  Kalman  filter  state  estimates  in  Figure  9. 

After  applying  the  Kalman  Filter  to  the  simulation  output  data,  the  estimates  of  the 
filter-design  state  vector,  the  residual  vectors,  and  also  the  covariance  matrices  associated 
with  these  vectors  are  available.  In  addition,  the  covariance  over  time  or  the  autocovariance 
matrix  Pxx(^i,t2)  is  also  available  [57:166]: 


Pxx(ti,t2)  —  ^Pxx(^i>^i)  —  ^P 


This  additional  information  may  be  used  to  improve  simulation  output  analysis  techniques. 

S.S.5  Kalman  Filter  Optimality.  Under  the  following  assumptions,  Kalman  filter 
state  estimates  are  guaranteed  to  be  optimal  by  essentially  any  logical  criteria,  such  as 
any  symmetric  cost  function  [57:231-236]. 
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•  The  dynamics  equation  is  a  linear  stochastic  differential  equation,  and  the  matrices 
F(<)  and  G(t),  or  equivalently  $(<„+!  jtn)  and  Gci(t),  are  known.  The  measurement 
model  is  linear,  and  the  matrix  H(t„)  is  known. 

•  The  dynamics  driving  noise  is  Brownian  motion  with  known  strength  Q(tn),  or  equiv¬ 
alently,  the  discrete-time  white  dynamics  noise  Wd(t„)  has  a  known  covariance  matrix 
Q<i(tn)-  The  measurement  noise  is  uncorrelated  normally-distributed  random  sequence 
with  known  covaricince  matrix  R(f„). 

•  The  initial  state  estimates  are  either  known  or  normally  distributed  with  a  known 
mean  Xq  and  covariance  matrix  Pq. 

Since  Brownian  motion  has  normally-distributed  independent  increments,  all  of  the  random 
variable  inputs  are  known  or  normally  distributed.  In  addition,  since  the  stochastic  differ¬ 
ential  equation  is  linear,  all  of  the  state  estimates  are  linear  combinations  of  jointly  normally- 
distributed  random  variables.  Therefore,  all  of  the  state  estimates  are  also  normally-distributed 
random  variables. 
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The  advantage  of  working  with  normally-distributed  random  variables  is  that  the  odd 
central  moments  beyond  the  first  one,  the  mean,  are  all  zero.  For  example,  the  third  moment 
which  measures  skewness  is  zero  since  the  normal  distribution  is  symmetric.  In  addition, 
the  even  central  moments  are  all  functions  of  the  second  central  moment,  the  variance. 
Therefore,  the  first  two  central  moments,  the  mean  and  the  variance,  completely  specify  the 
normal  probability  distribution. 

Furthermore,  the  mean,  mode,  and  median  are  all  the  same  point  for  the  normal  dis¬ 
tribution.  Since  most  of  the  optimality  criteria  select  one  of  these  points  as  the  optimal 
estimate,  under  the  above  assumptions,  the  Kalman  filter  estimates  are  optimal  by  essen¬ 
tially  all  meaningful  criteria.  According  to  Maybeck  [57:232],  the  Kalman  filter  state  estimate 
x(t+)  is  the  optimal  Bayesian  estimate  because  it  is  the  mean  of  the  probability  distribu¬ 
tion  of  the  state  conditioned  on  the  measurements,  /x(ti)|Z(t<)(dZj,  where  is  the  entire 
measurement  history  available  at  time  t,.  “By  virtue  of  being  the  conditional  mean,  x(t7 )  is 
also  the  minimum  mean  square  error  (MMSE)  estimate  [57:232].”  In  addition,  by  being  the 
conditional  mean  of  a  symmetric  distribution,  x(tf )  minimizes  any  symmetric  cost  function 
criteria  [57:232].  While  the  Bayesian  estimate  is  the  conditional  mean  of  /x(tj)lZ(ti)(^|2i), 
the  mode  of  this  distribution  is  called  the  maximum  a  posteriori  (MAP)  estimate.  Similarly, 
the  classical  maximum  likelihood  estimate  (MLE)  is  the  mode  of  /z(ti)|x(«<)(^»IO  [57:235]. 
Besides  being  the  MAP  estimate,  “x(t^)  is  the  maximum  likelihood  estimate  (MLE)  of 
x(t,)  if  Po  =  ool,  i.e.  if  Pq  ^  =  0,  and  converges  asymptotically  to  the  MLE  if  Pq  ^  0.” 

Maybeck  [57:235]  further  states  that  x{tf)  is  the  minimum  variance  unbiased  linear  estimate 
even  if  the  normal  distribution  assumption  is  removed  from  all  noise  inputs. 

3-4  System  Identification 

Before  applying  a  Kalman  filter  to  simulation  output,  the  system  dynamics  and  mea¬ 
surement  equations  must  be  determined.  For  engineering  applications,  these  equations  are 
developed  by  aggregating  the  effect  of  subsystem  components  and  empirical  testing.  Since 
discrete-event  simulations  are  used  in  applications  where  no  analytically  tractable  solution 
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exists,  the  dynamics  and  measurement  equations  must  be  deduced  from  the  output  sequence. 

Assume  the  discrete  stochastic  process  of  simulation  output,  which  has  attained  a 
stationary  steady-state  distribution,  can  be  represented  by  a  linear  time-invariant  discrete 
system  with  normally-distributed  noise  inputs.  The  time-invariant  version  of  the  dynamics 
model.  Equation  (19), 

x(i„)  =  ^x(t„-i)  -h  GdWd(t„)  (33) 

is  determined  by  the  constant  transition  matrix  #,  the  constant  dynamic  noise  input  matrix 
Gj,  ajid  the  time-invariant  covariance  matrix  Qd  of  the  discrete  white  normally-distributed 
zero-mean  dynamics  driving  noise  Similarly,  the  time-invariant  version  of  the  mea¬ 

surement  or  observation  model.  Equation  (20), 

z(t„)  =  Hx(t„)  -I-  v(t„)  (34) 

is  determined  by  specifying  the  constant  measurement  matrix  H  and  the  constant  covari¬ 
ance  matrix  R  of  the  discrete  white  normally-distributed  zero-mean  measurement  noise 
v(t„).  Therefore  to  specify  the  system  equations  completely,  G,  Qj,  H  and  R  must  be  es¬ 
timated.  However,  due  to  this  estimation,  this  Keilman  filter  application  is  no  longer  linearly 
independent  of  the  observations  and  optimality  cannot  be  guaranteed. 

Four  potential  approaches  are  considered  to  determine  the  appropriate  system  equation 
and  estimate  the  necessary  parameters.  First,  a  range  of  structural  forms  and  parameteriza- 
tions  can  be  evaluated  by  the  maximum  likelihood  estimation  (MLE)  procedure  developed 
by  Akaike  [6,  4,  5,  7,  8].  Second,  multiple  model  adaptive  estimation  (MMAE)  [48,  49,  58,  60] 
can  be  applied  with  various  classes  of  models.  Third,  transfer  functions  can  be  developed 
to  relate  past  observations  to  future  observations.  Finally,  the  structure  of  the  model  can 
be  imposed  by  a  priori  considerations,  and  the  necessary  parameters  can  be  estimated  by 
least  squares,  MLE  or  MMAE.  Each  of  these  four  approaches  to  determine  the  structure  will 
be  presented  in  the  following  sections.  However,  only  the  a  priori  formulation  is  selected, 
applied,  and  tested. 
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3.4-1  Akaike’s  Estimation  Methodology.  Several  researchers  demonstrate  the  equiva¬ 
lence  of  some  common  types  of  model  formulations.  Akaike  [4]  proves  that  there  is  no  differ¬ 
ence  between  a  Markovian  representation  of  an  output  sequence  from  a  stationary  stochastic 
process  and  an  autoregressive  and  moving  average  (ARMA)  representation.  Cooper  and 
Wood  [16]  show  equivalence  between  ARMA  and  state-space  representation  without  control 
inputs.  Hence,  if  an  ARMA  model  can  be  fit  to  simulation  output,  an  equivalent  state-space 
model  can  be  developed.  The  reverse  may  not  be  true  since  state-space  formulations  allow 
for  measurement  noise  which  is  not  included  in  ARMA  models. 

While  engineering  models  are  built  using  known  relationships,  time  series  models  are 
estimated  typically  from  the  realized  sequence.  Box  and  Jenkins  [11]  present  a  method¬ 
ology  to  fit  ARMA  models  to  univariate  data.  Two  limitations  in  their  methodology  are 
that  subjective  assessments  are  require!  and  that  multivariate  extensions  are  difficult  [8]. 
Overcoming  the  multivariate  limitation,  Charnes  [12]  extends  the  use  of  time  series  models 
for  simulation  output  analysis.  He  demonstrates  that  vector  autoregressive  moving  aver¬ 
age  (VARMA)  models  of  specific  forms  can  be  used  to  improve  the  estimation  of  combined 
confidence  regions  for  simulation  system  parameters.  Charnes’  results  demonstrate  that 
simulation  output  sequences  can  be  successfully  modeled  with  time  series  formulations,  and 
hence,  also  with  state-space  models. 

Akaike  [4]  proposes  the  following  meiximum  likelihood  procedure  for  stationary  normally- 
distributed  processes  with  a  Markovian  representation.  Markovian  representation  ensures  a 
finite  vector  representation.  First,  the  multivariate  normal  probability  density  function  is 

no  =  -  p]} 

where  ^  is  a  realization  of  dim  rsion  S,  //  is  the  mean  vector,  and  E  is  the  covariance  matrix. 
Since  the  natural  logarithm  is  a  strictly  monotonic  increasing  function,  the  maximum  of 
the  logarithm  of  a  function  is  the  same  as  the  maximum  of  the  original  function.  Letting 
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L  =  /(0>  natural  logarithm  of  the  likelihood  equals 
ln(i)  =  -|  ln(2x)  -  5  In  in  -  il?  - 

Define  actual  residuals  Tn  as  the  measurements  z„  minus  the  predicted  measurements 
Hx(t-),  as  shown  in  Equation  (26).  Using  the  whiteness  of  the  residuals,  their  joint  den¬ 
sity  function  can  be  written  as  a  product  of  the  normally  distributed  marginal  densities  of 
the  individual  residuals.  Incorporating  the  actual  residuals  and  their  associated  covariance 
matrix,  the  natural  logarithm  of  the  likelihood  function  for  N  observations  {z„}  is: 

QJ\J  1  ^  1  ^ 

Mi)=  -VMS’')-  +  +  (38) 

^  ^  n=l  n=l 

where  S  is  the  dimension  of  the  measurement  vector 
N  is  the  number  of  observations 
H  is  the  measurement  matrix 

P“  is  the  covariance  matrix  of  the  state  estimates  prior  to  the  measurement  update 
R  is  the  covariance  matrix  of  the  measurement  noise 
r„  is  the  residual  vector  for  the  nth  observation 

For  each  set  of  parameters  considered,  the  data  must  be  processed  through  the  Kalman  filter 
because  the  likelihood  function  assumes  a  complete  sequence  of  Kalman  filter  state  estimates. 
This  likelihood  function  can  be  maximized  by  employing  a  nonlinear  optimization  routine 
where  ef'ch  evaluation  requires  processing  all  of  the  observations  through  a  Kalman  filter. 

The  formulation  can  be  varied  by  including  more  or  less  of  the  past  observations  in 
the  observation  vector.  For  a  specified  dimension,  the  matrices,  $,Gj,H,Qj,R,  which 
maximized  this  function  are  the  Maximum  Likelihood  Estimation  (MLE)  parameters.  MLE 
estimates  the  parameters  without  any  explicit  penalty  for  increasing  the  number  of  param¬ 
eters  by  using  a  larger  system-state  matrix.  In  contrast,  Akaike’s  [4,  8]  technique  adds 
a  penalty  for  using  a  system-state  vector  with  larger  dimensions  and  more  parameters  to 
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estimate.  Akaike  defines  his  information  criterion,  the  AIC,  as 


AIC  =  —  2  In  ( meiximum  value  of  the  likelihood  function ) 

-|-  2  ( number  of  independently  adjusted  parameters  within  the  model ) 

The  first  term  is  a  penalty  for  “badness  of  fit”  of  the  model,  and  the  second  term  penalizes 
increased  unreliability  for  additional  parameter  estimation  [4].  When  models  of  the  same 
dimension,  and  hence  the  same  number  of  parameters,  are  compared,  the  addition  of  a 
constant  does  not  change  the  minimum  of  the  AIC.  Therefore,  for  models  of  the  same 
dimension,  the  technique  results  in  the  classical  MLE  methodology.  For  models  of  different 
dimensions,  the  best  choice  of  model  is  the  one  that  minimizes  the  AIC.  Akaike  refers  to  the 
selected  model  as  the  minimum  information  criteria  estimate,  MAICE.  Akaike  summarizes 
the  value  of  his  procedure  as  follows: 


By  the  introduction  of  MAICE  the  problem  of  statistical  identification  is  for¬ 
mulated  explicitly  as  a  problem  of  estimation  and  the  need  of  the  subjective 
judgment  required  in  the  hypothesis  testing  procedure  for  the  decision  on  the 
levels  of  significance  is  eliminated  completely.  [5] 

Because  of  the  extensive  numerical  computations  necessary  to  search  all  potential  mod¬ 
els  of  various  dimensions  and  parameterizations  using  the  AIC,  Akaike  [4]  says  the  feasibility 
of  this  procedure  is  almost  entirely  dependent  on  a  good  initial  guess  at  the  dimension  of 
the  system  and  selecting  a  corresponding  basis  of  the  state  space.  He  further  states  that  de¬ 
termination  of  the  Markovian  or  ARMA  representation  proceeds  in  two  steps,  first  selection 
of  the  structure  and  then  determination  of  the  parameters. 

Akaike  [6]  further  recommends  that  canonical  correlation  analysis  for  the  process  can 
provide  an  initial  guess  of  the  structure  and  the  parameters  to  be  used  for  the  initialization 
of  the  maximum  likelihood  procedure.  Akaike  [7]  says  that,  for  two  sets  of  random  variables, 
one  composed  of  the  past  and  present  values  and  the  other  composed  of  the  present  and 
future  values,  the  two  sets  of  canonical  variables  with  positive  correlation  form  the  minimal 
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information  interface  between  the  past  and  the  future  of  the  process.  From  this  minimal 
information  interface,  the  dimension  of  the  transition  matrix  can  be  determined. 

Green  [28:261]  describes  the  objective  of  canonical  correlation  as  finding  the  linear 
composite  variables  for  each  of  two  sets  such  that  the  highest  possible  correlation  is  attained 
between  the  linear  composites.  Subsequent  linear  composites  are  selected  such  that  they 
are  uncorrelated  with  previous  linear  composites  and  the  highest  conditional  correlation 
is  attciined.  Each  successive  pair  of  composites  between  the  two  sets  exhibits  decreasing 
correlation. 

Let  X  represent  one  variable  set  with  an  associated  vector  of  weights  a.  A  new  set 
of  variables  x*  is  determined  by  a^x.  Similaxly,  the  other  set  of  variables  y  and  another 
vector  of  weights  b  defines  a  new  set  of  variables  y*.  The  objective  of  canonical  correlation 
is  to  specify  a  and  b  such  that  correlation  between  x*  and  y*  is  maximized.  A  standard 
multivariate  result  is  that  the  a  and  b  which  maximizes  the  correlation  between  the  two  sets 
are  the  solutions  to  the  eigenvalue  problems: 

-  AI)a  -  0  and  (E-iEyxS-^Sxy  -  AI)b  =  0 

where  £  are  the  respective  covariance  or  cross- variance  matrices,  and  the  largest  eigenvalue 
X  is  the  squared  canonical  correlation  coefficient  [18:341-342]. 

For  a  time-invariant  system,  Akaike  [6]  defines  a  realization  of  the  $  matrix  of  mini¬ 
mal  dimension  as  a  minimal  realization.  Akaike  [8:68-70]  presents  the  steps  for  determining 
the  minimal  realization  using  the  MAICE.  His  process  begins  by  developing  two  sets  of 
observation's  for  canonical  analysis.  As  he  processes  through  the  data,  the  p  past  vectors 
plus  the  present  vector  is  an  observation  in  the  first  set  of  variables.  In  addition,  at  each 
data  point,  the  present  vector  plus  the  p  future  vectors  is  the  corresponding  observation  in 
the  second  set.  The  process  is  repeated  with  increasing  p.  The  number  of  vectors  in  the 
spanning  set  of  canonical  variates  is  the  dimension  p  of  the  system.  This  p  is  equivalent  to 
the  number  of  nonzero  canonical  variates  [8:60],  and  the  resulting  model  is  equivalent  to  an 
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ARMA(p,p-  1)  model  [27:7-8].  Akaike’s  system  identification  process  iterates  between  min¬ 
imization  of  the  AIC  given  the  model  dimension  and  canonical  correlation  with  an  additional 
lagged  vector  [8:68-70]. 

Cooper  and  Wood  [17]  modify  Akaike’s  procedure  by  including  the  present  vector  in 
only  the  future  set  rather  than  both  the  past  and  the  future  sets.  Goodrich  and  Stellwa- 
gen  state  in  the  Forecastmaster  software  documentation  [27:7-9]  that  the  advantage  of  this 
modification  is  that  all  of  the  canonical  variates  can  be  identified  in  one  step  rather  than  in 
a  stepwise  procedure.  An  ARMA(p,p)  model  results,  and  the  parameters  can  be  estimated 
by  maximum  likelihood  techniques  using  a  nonlinear  optimization  routine  such  as  Gauss- 
Newton.  The  parameter  estimates  which  are  small  compared  to  their  standard  error  are  set 
to  zero.  Porter  [76]  uses  Forecastmaster  [27]  in  a  feasibility  study  of  applying  the  Kalman 
filter  to  simulation  output.  He  shows  that,  for  waiting  times  from  an  MfMfl  queue,  a  two 
state  Kalman  filter  is  specified.  Tsay  and  Tiao  [99]  and  Tsay  [98]  improve  Akaike’s  method¬ 
ology  by  eliminating  the  requirement  for  the  canonical  analysis  step  to  specify  models  only 
of  order  ARMA(p,p). 

Schwarz  [89]  recommends  a  similar  procedure  except  that  Akiake’s  information  criteria 
(AIC)  is  replaced  with  a  Bayesian  information  criteria  (BIC): 

BIC  =  —  2  ln(  maximum  value  of  the  likelihood  function  ) 

-f  (  number  of  independently  adjusted  parameters ) 

This  technique  selects  the  model  that  is  a  posteriori  most  probable.  Quantitatively  both 
procedures  apply  the  principle  of  parsimony  [11:17-18]  in  model  building.  The  Schwarz  tech¬ 
nique  leans  toward  a  lower-dimension  model  [89].  Neftci  [72]  makes  a  comparison  between 
the  models  resulting  from  using  the  AIC  and  the  BIC  for  economic  time  series.  Applying 
both  criteria  to  eight  economic  time  series,  Neftci  fit  univariate  ARMA  models.  In  every 
case,  Schwarz’s  BIC  specifies  a  lower-dimension  model.  Neftci  reports  tha+  the  AIC  selected 
models  with  over  nine  parameters  on  average,  whereas  the  BIC  choose  models  which  average 
less  than  five  parameters. 
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In  concluding  this  section,  Akaike’s  procedure  or  one  of  the  derivatives  is  a  possibility 
for  identifying  a  dynamics  model  and  a  measurement  equation  for  simulation  output.  How¬ 
ever,  due  to  the  computational  requirements  to  evaluate  each  set  of  parameters  to  determine 
the  MLE,  this  approach  does  not  appear  well  suited  for  this  application. 

3.4  2  Multiple  Model  Adaptive  Estimation  (MMAE).  Another  potential  method  for 
system  identification  is  Multiple  Model  Adaptive  Estimation  [34,  48,  49,  58,  60,  61,  62, 
63].  MMAE  simultaneously  estimates  uncertain  filter-design  structural  parameters  and  state 
estimates.  MMAE  basically  approximates  the  unknown  parameter  space  with  discrete  points 
and  runs  a  Kalman  filter  at  each  discretized  point  in  parameter  space.  The  MMAE  estimation 
is  accomplished  by  monitoring  the  residuals  for  each  of  the  Kalman  filters  in  the  bank. 

In  contrast  to  MMAE,  the  unknown  state  parameters  can  be  augmented  to  the  state 
vector  and  estimated  along  with  the  states  using  a  nonlinear  estimation  technique  [58:Chapter 
12].  While  a  nonlinear  approach  requires  approximations,  Lainiotis  [49]  summarizes  the  re¬ 
lationship  between  the  nonlinear  estimation  of  the  state  estimate  at  time  t  conditioned  on 
the  available  measurement  history,  x(<|t,to)  in  his  notation,  and  MMAE  as  follows: 

The  optimal  nonlinear  filter  constitutes  an  exact  decomposition  or  partitioning  of 
the  nonlinear  filter  for  x(t]t,  to)  into  a  set  of  much  simpler  /mear  elemental  filters, 
namely,  the  Kalman-Bucy  filters,  which  are,  moreover,  completely  decoupled  {rom 
each  other.  [49] 

In  addition,  MMAE  can  be  used  to  determine  the  necessary  dimension  of  the  filter-design 
state  vector  [48]. 

Besides  determining  the  system  dimension,  MMAE  can  be  used  to  estimate  unknown 
parameters.  Let  a  denote  a  vector  of  the  parameters  to  be  estimated.  The  continuous  range 
of  values  for  a  are  discretized  into  L  representative  sets  of  values,  a/.  The  discretization  of 
a  can  be  thought  of  as  a  “grid” . 

The  discretization  typically  is  determined  by  the  effect  of  varying  the  particular  pa¬ 
rameter.  The  generalized  ambiguity  function  is  defined  as  the  average  value  of  the  likelihood 
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function  upon  which  the  estimator  of  a  is  based,  and  it  can  provide  insight  into  determining 
the  appropriate  number  of  discrete  levels  needed  for  unknown  parameters.  The  generalized 
ambiguity  function  is  given  by  Maybeck  [58:97-99]  where  6  is  some  function  of  the  estimated 
variables  and  Ci  represents  a  realization  of  the  measurement  history 

Let  L[6{ti),Ci]  denote  that  likelihood  function,  and  then  the  ambiguity  function 
^(■,  •)  is  defined  as  the  scalar  function  such  that 

A{6,e,)  =  r  ■■■  r  )(£.!«■ 

J —oo  */— oo 

where  6  is  some  value  of  the  estimated  variables  at  time  U  and  6t  is  the  true,  but 
unknown,  value  of  these  quantities. 

The  curvature  of  the  ambiguity  function  in  the  vicinity  of  the  true  parameters  indicates  the 
preciseness  with  which  the  maximum  likelihood  estimate  can  be  discerned  [57:97].  If  the 
ambiguity  function  has  little  curvature,  meaning  it  is  relatively  flat,  the  parameter  effect  is 
not  easily  discernible,  and  therefore,  a  coarse  grid  for  that  parameter  ought  to  be  sufficient. 
In  contrast,  if  the  ambiguity  function  has  a  distinct  peak,  small  variations  in  the  parameter 
should  be  discernible  in  the  magnitude  of  the  residuals.  Thus,  a  relatively  fine  discretation  is 
appropriate.  In  order  to  evaluate  the  ambiguity  function,  a  sensitivity  analysis  is  conducted 
[57:325-341].  In  this  analysis,  the  “truth  model”  is  a  detailed  model  with  assumed  true 
parameters.  Unlike  many  engineering  applications,  most  discrete-event  simulations  do  not 
provide  an  adequate  “truth  model”. 

After  the  discretization  of  the  parameter  space  is  complete,  let  the  probability  that 
the  unknown  set  of  parameters  a  assumes  the  set  of  L  discrete  values  a/  conditioned  on  the 
measurement  history  up  to  time  be  pj(tn)  =  Prob  (a  =  aJZ(t„)  =  Z„).  Assuming  the 
possible  values  for  a  are  limited  to  the  L  discrete  parameter  vectors  a/,  the  probability  is 
calculated  [61]  as 

/a!(tn)|a.Z({n_i)(^n|^j)  Zn-l  )  •  ) 


Pj(^n)  = 
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The  Bayesian  minimum  mean  squared  error  state  estimate,  the  sum  of  the  L  discrete  state 
estimates  weighted  by  their  associated  probabilities,  is  the  MMAE  estimated  state  vector: 

x(0  =  •?/(<„)  (37) 

i=i 

Each  is  calculated  using  a  discrete- time  Kalman  filter  with  the  associated  a;  pa¬ 

rameters.  With  time-invariant  matrices,  the  probabilities  are  calculated  using  the  residuals 
rj(tn)  —  —  HjX/(t„)  and  their  covariance  matrices  A/  =  +  R/.  Since  these 

residuals  are  jointly  normaJly  distributed, 

/a(tn)l«.Z(‘n-i)(*nla,,Z„_i)  =  (27r)-^/^lA,!-^/=*exp|-^rf(i„)Ar^r,(in)|  (38) 

where  S  is  the  dimension  of  the  measurement  vector  z„.  Since  the  residual  rj(<„)  for  the 
“best”  set  of  parameters  aj  should  be  relatively  small,  the  “best”  set  of  parameters  is  as¬ 
signed  a  high  probability  by  the  preceeding  pi{tn)  computation.  Similarly,  the  residual  for  a 
“mismatched”  model  should  be  relatively  large  and  the  associated  probability  is  small  [60]. 

May  beck  [58:131]  derives  the  covariance  matrix  for  the  random  vector  of  the  updated 
MMAE  state  vector  x(t^)  as 

p(<:) = i;B(<n){Pi(t:) + wo  -  ^(oiwc)  -  xcoi’'} 

i=\ 

Similarly,  from  May  beck  [58:132-133],  the  conditional  mean  for  the  unknown  vector  of  system 
parameters  a  at  time  t„  is 

a(t„)  =  E{a  -  a(t„)|Z(f„)  =  Z„}  =  aj  ■  pi(t„)  (39) 
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and  an  indication  of  the  precision  of  the  estimate  a(t„)  is  given  by  the  conditional  covariance 
matrix  of  a; 

S.|Z.  =  E{[a  -  a(i,)lla  -  a(i„)]’'|Z(t„)  =  Z„}  =  X;|a,  -  a(i„)![a,  -  p,(<„)  (40) 

i=l 

In  the  recursive  calculations  of  pi(tn),  if  any  of  the  pi[in)  becomes  zero,  it  remains 
zero  thereafter.  To  prevent  a  set  of  parameters  from  getting  discarded  prematurely,  the 
probabilities  typically  are  given  a  lower  bound  which  depends  on  the  number  of  discrete 
points  in  the  parameter  space  [60].  This  lower  bound  permits  the  parameters  to  continue  to 
adapt  throughout  the  observation  set. 

In  contrast  to  using  the  Bayesian  probabilistic  weighted  average  of  each  filter,  Maybeck 
and  Stevens  [63]  test  two  other  techniques  for  determining  the  state  estimate.  The  first 
alternative  is  the  Maximum  A  Posterior  (MAP)  version  of  MMAE,  which  simply  uses  the 
state  estimate  of  the  filter  with  the  highest  computed  probability.  This  prevents  including 
the  weighted  average  of  the  filters  with  probabilities  at  the  lower  bound.  A  middle  ground 
between  the  Bayesian  and  MAP  approach  is  to  include  within  the  weighted  average,  only 
those  filters  with  probabilities  over  a  certain  threshold. 

The  second  alternative  proposed  by  Maybeck  and  Stevens  [63]  is  to  modify  the  prob¬ 
ability  calculation.  They  note  that  the  leading  term  before  the  exponential  term  in  the 
probability  density  function,  Equation  (38),  scales  the  area  under  the  curve  to  one.  In  the 
case  where  the  residuals  r<  equally  match  their  associated  covariance  matrix  A/  (meaning 
the  rf{tn)Af^ri{tn)  are  equal),  the  respective  likelihoods  should  be  equal.  However,  the 
leading  scalar  term  unjustly  assigns  higher  probabilities  to  the  filters  where  the  determinant 
of  the  covariance  matrix  is  smaller.  Therefore,  Maybeck  and  Stevens  [63]  propose  removing 
that  leading  coefficient.  Since  each  probability  Pj{t„)  is  calculated  by  dividing  by  the  sum 
of  all  the  numerator  expressions  for  each  possible  value  of  aj,  each  probability  is  maintained 
between  zero  and  one,  and  the  sum  of  the  L  probabilities  equals  one. 

If  several  parameters  are  unknown,  the  number  of  Kalman  filters  running  in  parallel, 
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even  for  a  coarse  grid  of  possible  parameter  values,  may  be  prohibitive.  To  circumvent  this 
problem,  Maybeck  and  Hentz  [34,  61]  propose  applying  a  bank  of  Kalman  filters  for  which 
the  filter  parameters  are  dynamically  redeclared.  Thus,  the  assumed  parameter  values,  upon 
which  the  filters  in  the  bank  are  bcised,  move  through  the  feasible  parameter  space.  The 
small  bank  of  Kalman  filters  searches  for  the  best  parameter  estimates  by  redeclaring  each 
filter’s  pajameters  aj.  When  the  true  parameter  vector  appears  to  be  outside  the  current 
range  of  the  parameters  in  the  filter  bank,  one  of  two  options  is  employed.  The  assumed  filter 
parameters  in  the  bank  are  either  moved  or  their  range  expanded.  As  the  bank  converges  to 
“good”  estimates  of  the  parameters,  the  range  of  parameters  in  the  bank  of  filters  may  be 
contracted. 

Many  logical  schemes  for  dynamically  redeclaring  the  parameters  in  the  bank  of  Kalman 
filters  are  possible.  Maybeck  and  Hentz  [61]  test  several  rules  and  report  that  two  rules, 
which  position  the  center  of  the  Kalman  filter  bank,  worked  well.  The  first  rule  maintains 
the  probabilistic  weighted  average-  of  the  parameters  a(t„)  from  Equation  (39)  near  the  cen¬ 
ter  of  the  parameters  in  the  bank,  and  the  other  rule  positions  the  parameters  a/  with  the 
highest  probability  pi(tn)  as  the  center.  When  the  parameters  in  the  filters  are  redeclared, 
the  filters  with  new  parameters  are  initialized  with  the  probabilistic  weighted  average  of  the 
state  estimate.  Equation  (37).  In  addition,  all  the  filters  with  new  parameters  are  assigned 
an  equal  share  of  the  probability  from  the  filters  with  discontinued  parameters. 

Maybeck  and  Hentz  [61]  also  propose  a  contraction  criterion  and  an  expansion  crite¬ 
rion  for  the  range  of  parameters  in  the  Kcdman  filter  bank.  They  suggest  contracting  when 
a  matrix  norm  of  the  conditional  covariance  matrix  for  a(t„),  Sa|z„,  falls  below  a  certain 
threshold.  Their  expansion  criterion  is  based  on  residual  monitoring  using  the  filter  com¬ 
puted  residual  covariance  matrix,  Aj"^  =  [H/(t„)P/(<“)Hf  (f„) -f  Rj(f„)]“\  Specifically,  their 
criterion  is  that  if  all  the  likelihood  quotients 

Li(tn)  rf 
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are  large  and  close  in  magnitude,  the  bank  of  filters  is  expanded.  Both  of  these  rules  contract 
or  expand  the  filters  in  all  dimensions  parameter  space  at  the  same  time.  More  complicated 
approaches  may  contract  or  expand  only  in  specific  dimensions  of  the  unknown  parameter 
vector.  An  approach  which  expands  and  contracts  in  different  directions  would  be  useful  to 
hone  in  on  easily  estimated  parameters  while  continuing  the  coarse  search  for  other  param¬ 
eters  that  are  more  difficult  to  estimate. 

Harrison  and  Stevens  [31,  32]  apply  an  approach  similar  to  MMAE  to  estimate  time 
series  data.  Their  technique  uses  a  bank  of  four  Kalman  filters.  Each  filter  is  designed  for 
a  different  condition;  steady-state,  transient  value,  slope  change,  or  jump  change.  At  each 
stage,  the  sixteen  probabilities  of  transitioning  from  each  filter  to  any  filter  are  determined. 
To  avoid  a  growing  bank  of  filters,  the  probabilities  and  weighted  state  estimates  are  used 
to  reduce  back  to  the  original  four  filters. 

Like  Akaike’s  method,  the  MMAE  technique  is  another  alternative  approach  to  deter¬ 
mine  the  system  structure.  MMAE  may  also  be  used  as  a  method  to  estimate  unknow  system 
parameters.  The  MMAE  technique,  particularly  with  the  use  of  the  moving  bank  of  filters, 
appears  to  require  considerably  less  computation  than  Akaike’s  technique.  However,  many 
filters  may  be  required  to  span  different  structural  models  each  with  unknown  parameters. 

S.4.S  Transfer  Function  Technique.  Copper  and  Wood  [16]  relate  the  state-space 
representation  to  a  transfer  function  model.  The  transfer  function  is  a  linear  operator  which 
relates  the  system  inputs  to  outputs  [11].  For  example,  the  Autoregressive  Moving  Average 
(ARM A)  time  series  models  are  equivalent  to  types  of  transfer  functions  which  relate  a 
normally-distributed  white  noise  input  to  an  output  sequence.  Given  a  stream  of  output 
data,  the  transfer  function  can  be  estimated  either  in  the  time  or  frequency  domains.  In 
the  time  domain,  an  ARMA  model  is  fit,  whereas  in  the  frequency  domain,  the  model  is 
estimated  by  using  a  Fourier  transformation.  Lee  [54:99-107]  describes  a  methodology  to 
indentify  the  transition  matrix  ^  by  transforming  to  canonical  form  and  estimating  the 
parameters  with  least  squares. 


73 


S.4-4  ^  Priori  Model  Formulation.  For  specific  applications,  the  state-space  struc¬ 

ture  may  be  imposed  on  the  simulation  output  sequence  by  a  priori  considerations.  Under 
these  circumstances,  the  dimension  and  structure  of  the  problem  are  fixed,  and  the  param¬ 
eters  caji  be  estimated  by  least  squares,  Meiximum  Likelihood  Estimation  (MLE),  Multiple 
Model  Adaptive  Estimation  (MMAE)  [58],  or  correlation  techniques  [9,  65,  66].  Harvey 
[33]  presents  the  necessary  likelihood  functions  for  several  structures  which  are  commonly 
encountered  in  time  series  data. 

3.5  Summary 

The  first  major  section  of  this  chapter  reviews  the  theory  of  stochastic  processes.  A 
discrete-time  dynamics  relationship,  Equation  (19),  and  a  measurement  model,  Equation  (20) 
are  shown.  Using  these  system  equations,  the  next  section  developes  the  discrete-time,  time- 
invariant,  linear,  Gaussian,  Kalman  filter  equations.  The  final  section  in  the  chapter  describes 
four  potential  techniques  for  determining  and  estimating  the  filter-design  system  equations 
for  simulation  output  data.  The  next  chapter  applies  these  concepts  to  the  specific  problem 
of  simulation  steady-state  identification. 
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IV.  Single-Run  Steady-State  Identification:  Residual  Monitoring  Approach 


4 . 1  Introduction 

In  discrete-event  simulation,  a  typical  output  analysis  objective  is  to  estimate  the  char¬ 
acteristics  of  the  model  outputs  of  interest,  such  as  the  mean  or  variance.  One  class  of  models 
is  that  of  infinite-horizon  simulations,  in  which  our  interest  would  be  in  estimating  steady- 
state  parameters.  Although  it  is  possible  that  these  parameters  may  be  cyclic,  this  research 
focuses  on  output  sequences  that  attain  a  stationary  steady-state  probability  distribution. 

Typically,  a  simulation  model  is  not  initialized  at  steady-state  conditions.  The  initial 
transient  or  startup  problem  arises  from  the  fact  that  the  model’s  output  may  initially  pass 
through  an  unrepresentative  phase  as  it  approaches  steady  state.  Including  transient  data 
in  output  analysis  may  bias  steady-state  estimates,  as  described  in  Chapter  II.  A  generally 
accepted  solution  is  to  truncate  the  output  sequence,  thereby  eliminating  the  initial  data 
that  may  bias  the  steady-state  estimates.  While  initial  data  truncation  may  increase  the 
mean  squared  error  of  the  point  estimator  [21,  104],  deletion  improves  confidence  interval 
coverage  markedly  [44].  This  chapter  proposes  and  tests  an  analytical  approach  to  determine 
the  appropriate  initial-data  truncation  point. 

Selection  of  the  appropriate  truncation  point  may  be  obfuscated  by  the  difficulty  that 
the  transient,  rather  than  ending  at  a  particular  point,  may  converge  gradually  to  the  steady- 
state  distribution  [101].  The  effective  end  of  the  transient  Uo  is  the  point  after  which  the 
initial  conditions  no  longer  have  a  significant  effect  on  the  estimated  performance  measures 
[15].  Deleting  the  data  prior  to  reduces  the  bias,  but  deleting  data  after  tIq  unnecessarily 
reduces  the  sample  size  and  increases  the  variance  of  the  estimates.  The  specific  problem 
addressed  by  this  chapter  is  estimating  an  appropriate  truncation  point  ho  for  a  univariate 
simulation  output  sequence  {yi,y2,  ■  ■  ■  ,yN}-  For  ^^he  confidence  interval  construction  tech¬ 
niques  designed  for  one  long  simulation  run,  such  as  batch  means,  this  truncation  point 
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identification  algorithm,  which  also  works  on  a  single  replication,  is  a  useful  complementary 
technique. 

The  background  on  related  discrete-event  simulation  research  is  presented  in  Chap¬ 
ter  II,  and  Chapter  III  describes  the  Kalman  filter.  This  chapter  is  divided  into  four  major 
sections.  The  proposed  system  models  and  a  model  estimation  strategy  is  developed  in  the 
first  two  sections.  The  next  section  on  the  truncation  point  selection  methodology  discusses 
Kalman  filter  residual  monitoring,  and  the  following  section  presents  the  steps  in  the  pro¬ 
posed  algorithm.  The  results  section  reports  on  the  Monte  Carlo  analyses  of  second-order 
autoregressive  AR(2)  data  and  first-order  autoregressive-moving  average  ARM A(  1,1)  data. 
Although  the  residual  monitoring  approach  does  not  perform  as  well  as  the  MMAE  approach 
in  Chapter  V,  Section  4.2  provides  the  model  formulation  used  in  all  of  the  subsequent  meth¬ 
ods. 

4-8  Determination  of  the  System  Structure 

The  first  step  necessary  in  applying  a  Kalman  filter  is  to  determine  appropriate  system 
equations.  Four  techniques  are  presented  in  the  last  heilf  of  Chapter  III.  Akaike’s  minimum 
information  criterion  (MAICE),  the  Multiple  Model  Adaptive  Estimation  (MMAE),  and 
transfer  functions  can  be  used  to  determine  appropriate  system  equations  from  general  classes 
of  possible  systems.  In  addition,  the  a  priori  method  uses  additional  information  to  limit 
the  possible  structure  of  the  dynamic  and  measurement  equations.  Limiting  the  possible 
structure  and  parameters  significantly  simplifies  the  system  identification  task.  Kelton  and 
Law  [44]  and  Schruben  [87]  impose  a  structure  on  the  simulation  output  and  achieve  good 
results.  Therefore,  using  a  similar  structure  for  the  form  of  the  system  equations,  an  a  priori 
approach  is  developed.  The  first  subsection  presents  an  a  priori  model  formulation.  The 
associated  Kalman  filter  equations  and  relationships  are  developed  in  the  second  subsection. 

4-8.1  A  Priori  Model  Formulation.  In  order  to  apply  the  Kalman  filter  to  simulation 
output,  an  appropriate  dynamics  model,  Equation  (19)  on  page  49,  and  the  associated 
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measurement  model,  Equation  (20),  must  be  determined.  Since  steady-state  output  is  being 
modeled,  time-invariant  equations  (33)  and  (34)  on  page  62,  are  chosen. 

Abate  and  Whitt  [1],  Harrison  and  Williams  [30],  and  Whitt  [102]  use  regulated  Brow¬ 
nian  motion  as  the  underlying  model  in  analysis  of  queuing  systems.  A  similar  formulation 
was  considered  for  the  dynamics  equation.  However,  this  approach  was  rejected  because  of 
the  nonlinear  state  transitions  that  would  have  to  be  taken  into  account  in  the  Kalman  filter. 

Kelton  and  Law  [44],  as  shown  in  Equation  (13)  on  page  33,  and  Schruben  [87],  as 
shown  in  Equation  (7)  on  page  28,  suggest  modeling  the  steady-state  simulation  output 
as  a  constant  mean  plus  noise  T](tn)‘- 

y(tn)  =  Mv  +  ^(^n)  for  observations  n  =  1,2, . . . ,  N  (41) 

The  term  is  correlated  noise  with  E[T](tn)]  =  0.  These  researchers  attempt  to  account 
for  the  correlation  in  the  noise  term.  Kelton  and  Law  [44]  use  generalized  least  squares  rather 
than  ordinary  least  squares  to  account  for  the  correlated  noise  term.  Schruben  [87],  on  the 
other  hand,  transforms  the  sequence  so  that  the  noise  is  modeled  as  a  Brownian  bridge.  In 
both  applications,  the  underlying  model  of  the  steady-state  simulation  output  is  a  constant 
mean  plus  noise.  In  the  Kalman  filter  approach,  with  the  same  model  for  the  simulation 
output,  the  dynamics  and  measurement  models  must  account  for  the  correlation  in  the  noise. 

Using  the  same  underlying  model,  the  steady-state  simulation  output  sequence  {2/(in)} 
can  be  approximated  by  a  steady-state  mean  fiy  plus  a  time-correlated  noise  r]{tn).  The  se¬ 
quence  of  time-correlated  noise  is  approximated  as  the  output  of  a  linear  stochastic  dynamics 
system.  The  linear  modeled  noise  terms  rj{in)  and  their  lagged  values  are  the  filter-design 
state  vector  x(tn)  in  the  system  model.  A  similar  concept  is  used  by  Fishman  [20,  23]  to 
construct  confidence  intervals  for  an  estimate  of  the  process  mean.  Fishman’s  approach 
fits  an  autoregressive  time  series  model  to  the  simulation  output  and  uses  the  autoregres¬ 
sive  coefficients  to  estimate  the  variance  of  the  mean  estimate.  Fishman’s  technique  applies 
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time-invariant  autoregressive  coefficients.  Since  steady-state  output  is  being  modeled,  time- 
invariant  system  matrices  seem  reasonable. 

Modeling  the  steady-state  phase  of  simulation  output  by  means  of  a  linear  time- 
invariant  system  has  been  questioned.  In  evaluating  frequency  domain  experimentation, 
Saxgent  and  Som  [81]  state  that  the  input  and  output  process  cf  a  modulated  Af/M/1  simu¬ 
lation  does  not  constitute  a  time-invariant  linear  system.  However,  their  analysis  is  confined 
to  examining  the  simulation  model  from  inputs  to  outputs.  In  their  study,  the  input  is  the 
cosine  of  the  average  service  time  and  the  output  is  total  time  in  the  system.  In  contrast,  the 
time-series  approach  assumes  a  linear  time-invariant  system  between  lagged  output  values. 
Thus,  Sargent  and  Som’s  objections  do  not  apply  to  this  formulation. 

The  assumed  constant  autocorrelation  relationship  of  a  linear  time-invariant  system 
may  be  inappropriate.  Using  the  simulation  of  an  M/M/1  queue  as  an  example,  let  the 
output  of  interest  be  either  the  waiting  time  for  each  entity  or  the  total  number  in  the 
queue.  When  the  queue  length  is  long,  a  strong  autocorrelation  in  the  simulation  output 
seems  logical.  However,  observations  of  either  entity  waiting  times  or  queue  lengths  that 
are  separated  by  a  period  of  time  when  the  server  is  idle  are  functions  of  nonoverlapping 
sequences  of  interarrival  and  service  times.  Both  sequences  are  pseudorandom  numbers, 
which  are  designed  to  be  uncorrelated  in  time.  Thus,  when  separated  by  a  period  with  an 
idle  server,  no  apparent  autocorrelation  may  exist  in  the  output  for  either  waiting  times  or 
queue  lengths.  Furthermore,  the  degree  of  autocorrelation  may  increase  with  queue  length. 
In  spite  of  this  objection,  like  Fishman’s  autoregressive  time  series  approach  [20,  23],  this 
application  assumes  a  linear  time-inveiriant  system  between  lagged  simulation  output  val¬ 
ues.  The  approximation  ought  to  be  sufficiently  accurate  for  the  purpose  of  steady-state 
identification. 

An  appropriate  form  and  order  of  the  noise  model  must  be  determined  and  estimated. 
Odoni  and  Roth  [73]  investigate  the  simulation  state  of  queue  length.  They  state  that,  for 
queuing  systems  during  the  transient,  the  sequence  of  the  simulation  states  is  dominated  by  a 
decaying  exponential.  Similarly,  Conway  [15]  states  that  the  transient  components  typically 
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decay  geometrically.  While  Odon'  and  Roth’s  [73]  and  Conway’s  [15]  conclusions  apply  to  the 
transient  phase  of  simulation  output,  they  seem  to  indicate  that  the  dominant  autocorrelation 
in  the  output  sequence  can  be  approximated  by  relatively  simple  autoregressive  models. 

In  time  series  analysis,  a  decaying  exponential  is  modeled  as  a  first-order  autoregressive 
AR(1)  process  [11:57], 

Vitn)  =  +  Wd{t„)  (42) 

where  <f>i  is  the  first  autoregressive  coefficient  and  is  a  sequence  of  white  normally- 

distributed  noise  terms.  A  second-order  autoregressive  AR(2)  process, 

^(<n)  =  +  (f>2Vitn-2)  +  Wd{tn)  (43) 

can  model  a  mixture  of  damped  exponentials  and  a  sinusoidal  within  an  exponentially  de¬ 
caying  envelope  [11:59-60]. 

Steudel  and  Wu  [97]  and  Schriber  and  Andrews  [85]  report  that  periodic  samples  of 
M/M/1  queue  length  can  be  represented  adequately  as  an  AR(1).  Charnes  [12]  reports  that 
a  vector  of  queue  lengths  from  more  complex  simulation  models  can  be  modeled  adequately 
as  a  first-order  vector  autoregressive,  VAR(l),  model. 

In  this  research,  the  simulation  output  is  modeled  as  a  steady-state  mean  plus  corre¬ 
lated  noise.  The  true  time-correlated  noise  7/(tn)  is  approximated  with  autoregressive  noise 
fi(t„).  The  dynamics  of  the  correlated  noise  is  formulated  either  as  an  AR(1)  with  measure¬ 
ment  noise  model,  which  has  a  scalar  filter-design  state  i(t„),  or  as  an  AR(2)  with  mea¬ 
surement  noise  model,  which  has  a  two-dimensional  state  vector  x(t„).  Both  approaches, 
the  AR(1)  with  measurement  noise  and  the  AR(2)  with  measurement  noise  formulations, 
are  developed  and  tested  in  this  chapter.  While  the  noise  in  simulations  may  have  higher 
order  effects,  in  many  engineering  applications,  reduced  order  models  are  sufficient.  In  addi¬ 
tion,  the  “lack  of  fit”  of  these  simple  models  perhaps  can  be  approximated  as  measurement 
noise.  This  a  priori  model  of  a  steady-state  mean  plus  autoregressive  noise  is  imposed  on 
discrete-event  simulation  output.  The  next  subsection  presents  the  corresponding  Kalman 
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filter  equations. 


4.2.2  The  Kalman  Filter  Equations.  Autoregressive  models  can  be  written  as  dis¬ 
crete,  linear,  time-invariant,  stochastic,  dynamic  systems  with  the  appropriate  choices  of 
defining  matrices.  The  filter-design  system  states  x(t„)  are  the  autoregressive  noises  rj{t„), 
and  the  state  transition  matrix  $  relates  one  state  vector  to  the  next.  The  noise  disper¬ 
sion  matrix  Gj  determines  how  the  white  normally-distributed  scalar  dynamic  noise  uJd(t„) 
affects  the  states.  Assume  that  the  noise  correlation  can  be  approximated  with  an  autoregres¬ 
sive  model.  The  dynamics  model.  Equation  (33),  and  measurement  model.  Equation  (34), 
matrices  for  an  AR(1)  with  measurement  noise  formulation  are 

x(t„)  =  rj{tn),  ^  =  4>x,  Gd  =  1  and  H  =  1  (44) 


and  for  an  AR(2)  with  measurement  noise  formulation  are 


x(<n) 


^(in) 

^(in-l) 


$  = 


4>2 

1  0 


Gd  = 


and  H  = 


1  0 


(45) 


With  this  formulation,  wj{tn),  Qd{tn),  u(t„),  and  R(tn)  are  scalars.  The  vari¬ 

ances  of  the  discrete,  white,  normally-distributed,  zero-mean,  dynamic  noise  Wd{tn)  and 
measurement  noise  v{t„)  are  assumed  to  be  constant  for  the  steady-state  simulation  output. 
Thus,  Qd{in)  =  Qd  and  R(t„)  =  R,  respectively. 

Using  either  of  the  two  possibilities  for  H,  the  modeled  measurement  2(<n)  without  the 
noise  term  v{tn)  from  Equation  (34)  is 


Hx(t„)  =  rj{t„)  (46) 

The  actual  measurements  2„  of  the  correlated  noises  are  determined  from  the  realiza¬ 
tion  of  simulation  output  as  follows  from  the  random  variable  relationship  expressed  in 
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Equation  (41): 


(47) 


=  3/n  -  Mv 


Harvey  [33:52-53]  differentiates  between  the  pure  AR  models  without  measurement 
noise  and  the  AR  models  with  measurement  noise.  Since  the  state  is  unobserved  with 
measurement  noise,  Harvey  calls  the  Autoregressive-Integrated-Moving  Average  (ARIMA) 
models  with  measurement  noise  Unobserved  Components  ARIMA  (UCARIMA)  models.  A 
pure  AR(1)  process  has  theoretical  autocovariances  of  the  state  x(t„)  at  lag  r  of 

T  =  0, 1, 2, . . . 

1  -  <Pi 

and  has  an  autocorrelation  function  of 


for  r  =  0,1,2,... 


In  the  ARMA  case,  one  of  the  state  vector  elements  x(t„)  equals  the  observation  j/(in)  minus 
the  steady-state  mean  /Zy.  In  contrast,  the  states  in  the  UCARMA  formulation  are  observed 
with  measurement  noise  u(t„)  with  variance  R.  The  variance  of  cin  UCAR(l)  sequence 
includes  the  measurement  noise  variance, 


7(0)  = 


1  -  (i)\ 


+  R 


but  the  lagged  autocovariances  are  the  same  as  the  AR(1)  case  since  the  measurement  noise 
is  uncorrelated  in  time.  Therefore,  the  UCAR(l)  autocorrelation  function  is 


^v(^) 


<i>lQd{i  - 
Qd{i  -  +  R 


for  r  =  0, 1,2, . . . 


Inclusion  of  measurement  noise  requires  that  the  variance  R  of  the  measurement  noise  u(t„) 
must  also  be  estimated.  Rather  than  using  the  Harvey’s  UCAR(l)  and  UCAR(2)  notation, 
these  formulations  are  referred  to  as  the  AR(1)  with  measurement  noise  model  or  the  AR(2) 
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with  measurement  noise  model. 

Pure  AR  models  do  not  include  measurement  noise  but  they  can  be  thought 

of  as  a  degenerative  form  of  the  system  model  upon  which  Kalman  filter  is  based.  The 
degenerative  form  simply  requires  eliminating  the  measurement  noise  v(tn).  Setting  the 
variance  of  the  mecisurement  noise  to  zero,  iZ  =  0,  removes  the  measurement  noise  from 
the  system  model  since  it  edready  has  a  mean  of  zero.  For  the  above  autoregressive  models, 
applying  the  Kalman  filter  equations  with  R  =  0  results  in  the  same  estimates  as  applying 
a  pure  AR(1)  or  AR(2)  model. 

» 

A  simple  first  or  second  order  autoregressive  model  may  not  be  adequate  since  Fishman 
[20,  23]  requires  considerably  more  autoregressive  terms  in  his  applications  to  simulation 
output.  Specifically,  he  requires  four  lagged  values  to  model  the  output  of  an  M/M/1  queue 
[20],  seven  lags  for  a  simple  airline  reservation  problem  [23:256-257],  and  twenty-three  lagged 
values  for  a  cyclical  washer  problem  [23:259].  However,  these  system  equations  provide  two 
ways  to  account  for  “lack  of  fit’’  due  to  assuming  the  correlated  noise  can  be  modeled 
adequately  as  a  low-order  autoregressive  noise  Since  discrete-event  simulation  output 

yn  is  known  exactly,  the  variance  R  of  the  measurement  noise  u(<„)  may  represent  “lack 
of  fit”  between  the  true  noise  T/(tn)  and  the  assumed  autoregressive  noise  rj{tn).  Second, 
reduced  order  models  are  applied  in  many  engineering  applications  by  simply  increasing  the 
variance  Qj  of  the  dynamic  driving  noise  Wd{tn)  [57]. 

For  the  proposed  model  of  the  steady-state  simulation  output,  the  system  rriatrices 
G^,  Qd,  H  and  R  are  assumed  time-invaxiant.  Therefore,  the  Kalman  filter  ^ain  K(t„) 
and  state-estimate  variance  matrices  P(t")  and  P(t^)  attain  steady-state  values  as  dis¬ 
cussed  in  Section  3.3.4.  In  this  application,  the  steady-state  matrices  are  used  in  all  calcula¬ 
tions.  Since  steady-state  values  axe  applied  throughout,  the  time  subs';ripts  on  the  matrices 
K,  P-,  P+  are  not  necessary. 

This  a  priori  formulation  can  be  implemented  two  different  ways.  The  simulation  out¬ 
put  can  be  processed  directly  through  this  model  of  the  Kalman  filter  as  in  most  engineering 
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applications.  Another  approach  revolves  around  the  idea  that  the  simulation  output  can 
first  be  smoothed  by  a  moving  average  as  in  Welch’s  [101]  technique.  The  moving  average 
replaces  each  observation  with  the  average  of  a  predetermined  number  of  past  and  future 
observations.  This  moving  average  removes  some  of  the  high  frequency  oscillations  in  the 
output.  This  smoothed  data  then  can  be  processed  through  the  Kalman  filter  to  deter¬ 
mine  the  truncation  point.  The  approach  taken  in  this  chapter  is  to  process  the  simulation 
observations  directly. 

In  order  to  implement  this  formulation  of  the  Kalman  filter,  the  system  parameters 
must  be  estimated.  The  AR(1)  with  measurement  noise  formulation  has  four  unknown  pa¬ 
rameters:  the  steady-state  mean  fiy,  the  variance  Qd  of  the  dynamics  driving  noise  Wd{tn),  the 
variance  R  of  the  measurement  noise  u(t„),  and  the  autoregressive  coefficient  <l>i.  The  AR(2) 
with  measurement  noise  formulation  has  an  additional  unknown  autoregressive  coefficient  4>2 
to  estimate.  Their  estimation  is  discussed  in  the  next  section. 

4-S  System  Parameter  Estimation 

The  a  priori  formulation  proposed  in  the  previous  section  requires  the  system  pa  am- 
eters  to  be  estimated.  These  parameters  are  the  observation  mean  fXy,  the  autocorrelation 
coefficients  <^i  (and  <f>2  for  an  AR(2)  with  measurement  noise  formulation),  the  dynamic  noise 
variance  Qdi  and  the  measurement  noise  variance  R.  Several  estimation  procedures,  such 
as  Multiple  Model  Adaptive  Estimation  (MMAE),  Maximum  Likelihood  Estimation  (MLE), 
least  squares  estimation,  or  correlation  techniques  [66],  are  possible  approaches. 

In  the  method  of  MMAE  (See  Section  3.4.2),  a  bank  of  Kalman  filters,  each  with  a 
different  set  of  parameters,  is  run.  The  Maximum  A  Posterior  (MAP)  version  of  MMAE 
selects,  eis  estimates,  the  parameters  from  the  filter  with  the  most  probable  residuals.  The 
Bayesian  version  of  MMAE  uses  the  likelihood  of  each  filter  to  calculate  a  weighted  average 
for  the  parameter  estimates,  eis  shown  in  Equation  (39). 

Classical  MLE  selects  as  parameter  estimates  the  values  which  maximize  the  joint 
probability  of  the  actual  measurements.  Least  squares  estimation  picks  the  parameter  esti- 
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mates  that  minimize  the  sum  of  the  squared  residuals.  Therefore,  while  MLE  and  MMAE 
are  calculated  based  on  the  magnitude  and  assumed  probability  distribution  of  the  Kalman 
filter  residuals,  least  squares  is  based  only  on  the  size  of  the  residuals.  For  a  univariate 
sequence,  least  squares  estimates  are  generally  asymptotically  equivalent  to  MLE  [33:129]. 

The  correlation  technique  proposed  by  Mehra  [66]  matches  the  theoretical  autocorre¬ 
lations  for  the  assumed  model  with  the  sample  autocorrelations.  After  solving  for  the  mean 
and  autocorrelation  coefficients,  the  noise  variances  are  calculated  from  the  sample  variance 
of  the  resulting  residuals. 

This  section  discusses  the  identifiability  of  the  noise  variances,  Qd  and  R.  The  effect 
of  this  indeterminancy  between  Qd  and  R  on  the  various  estimation  approaches  is  discussed. 
In  order  to  eliminate  the  indeterminancy,  the  first  term  of  the  Kalman  filter  gain,  Ki  =  A:,  is 
estimated  instead  of  the  variances.  Because  of  the  lack  of  variance  information,  least  squares 
estimation  is  selected.  Using  the  estimated  system  parameters  and  the  residuals,  the  noise 
variances,  Qd  and  R,  the  state  estimate  variances,  P“  and  P"*",  and  the  variances  of  the 
output  and  the  mean  estimator  can  be  calculated. 

4.3.1  Identifiability.  A  set  of  parameters  is  identifiable  if  no  other  set  of  parameters 
has  the  same  joint  density  function  [33:205]. 


Identifiability  has  an  immediate  bearing  on  estimation.  If  two  structures  have 
the  same  joint  density  function,  the  probability  of  generating  a  particular  set  of 
observations  is  the  same  for  both  structures.  Thus  there  is  no  way  of  differenti¬ 
ating  between  them  on  the  basis  of  the  data.  Furthermore,  it  often  will  be  the 
case  that  attempts  to  estimate  models  which  are  not  identifiable  will  run  into 
practical  difficulties  [33:205]. 


Simultaneously  estimating  both  the  dynamics  noise  variance  Qd  and  the  measurement 
noise  variance  R  is  an  identifiability  problem.  As  discussed  on  page  55  and  in  Section  3.3.4, 
the  estimated  state  variances  are  not  effected  by  the  actual  measurements.  Further,  the 
solution  of  Equation  (32)  gives  the  steady  state  solution  for  P~.  If  the  noise  variances. 
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Qd  and  R,  are  multiplied  by  a  positive  constant,  P“  multiplied  by  the  same  constant  is  a 
solution  of  Equation  (32).  Using  the  scaled  variances  and  calculating  the  Kalman  filter  gain 
with  Equation  (30)  results  in  the  original  gain.  Since  the  system  matrices  Gj,  H,  K  along 
with  the  sequence  of  measurements  determine  the  residuals,  scaling  all  the  variances  by  a 
positive  constant  does  not  change  the  resulting  residuals.  For  scalar  system,  the  Kalman 
filter  gain  is  a  function  of  the  ratio  of  ^  [57:225],  so  any  combination  with  the  same  ^  ratio 
generates  the  same  residuals  and  has  equal  likelihood  of  being  the  correct  model.  Therefore, 
Qd  and  R  can  not  be  estimated  simultaneously. 

4-3.2  Estimation  Techniques.  In  spite  of  the  confounding  effect  described  in  the  pre¬ 
vious  subsection,  an  approach  for  making  unique  estimates  of  the  system  parameters  is 
developed  in  this  subsection.  The  estimation  approach  is  based  on  “concentrating  out”  ei¬ 
ther  Qd  or  R.  The  effect  of  this  approach  on  MLE,  MMAE,  and  least  squares  estimation  is 
discussed.  Rather  than  “concentrating  out”  Qd  or  R,  estimation  of  the  Kaman  filter  gain 
directly  is  examined  and  selected. 

For  a  univariate  sequence,  a  common  approach,  if  both  of  the  noise  variances  are 
unknown,  is  one  of  the  variances  should  be  “concentrated  out”  of  the  likelihood  equation. 
Harvey  [33:107,126]  shows  that  by  setting  one  variance  as  a  scalar  multiple  of  the  other 
variance,  one  of  the  variances  can  be  set  arbitrarily  and  the  other  can  be  estimated  using 
MLE  without  the  problem  of  indeterminancy. 

To  understand  Harvey’s  approach,  let  al  be  the  unknown  variance  to  be  “concentrated 
out”  of  the  likelihood  function.  Harvey  begins  by  dividing  all  of  the  variances  by  cr^.  Let 
=  ifP~  P^  ~  ~  R,  =  ^R.  The  same  Kalman  filter 

variance  propagation  and  update  equations  can  be  used,  and  the  original  Kalman  filter  gain 
K,  Equation  (30),  is  attained.  Harvey  shows  that  MLE  can  be  applied  with  these  scaled 
variances.  The  natural  logarithm  of  the  likelihood  function  for  a  multivariate  sequence  with 
time-invariant  system  matrices  is  Equation  (35)  on  page  64.  The  version  for  a  univariate 
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sequence  with  measurement  noise  with  steady-state  covariance  matrices  is 


ln(Z)  =  -^ln(27r)-  ^ln(HP-H^-^i?) 


T 


2 

n 


2(HP-H^  +  ii:) 


where  N  is  the  number  of  observations 
H  is  the  measurement  matrix 

P“  is  the  covariance  matrix  of  the  state  estimates  prior  to  measurement  update 
R  is  the  covariance  matrix  of  the  measurement  noise 
r„  is  the  residual  for  the  nth  observation 


Substituting  in  the  scaled  variances, 


ln(L)  =  -f  ln(27r)  -  f  alR,)  - 


=  -  f  ln(<rj)  -  f  ta(HP.-H’'  +  fi.)  - 


_ L^n=l  " 


HP.-Hr+R. 


Harvey  differentiates  with  respect  to  Since  the  measurement  sequence  and  the  Kalman 
filter  gain  are  unchanged  by  scaling  with  the  residuals  are  the  sar:i.e.  In  addition,  the 
residual  variance,  RP“R^  -(-  R»  is  not  a  function  of  cr^: 

^  _  A  ,  (  1  ^  rl  ^ 

dal  2al  [2(aiy )  RP,-R^  -f  R, 

Multipling  through  by  2(cr^)^  and  solving, 

2  _  AT  '  n 

•  ~  RP.-R^  +  R, 

The  second  partial  of  ln(Z/)  with  respect  to  al  is 

^  _JL-  _  (J  \  E:,.  rj  ^  ^  n 

avj  +  2<r; 

Since  al  is  a  variance,  it  must  be  non-negative.  Thus,  the  second  partial  of  ln(Z/)  is  strictly 
positive,  and  setting  the  first  partial  equal  to  0  yields  the  al  which  maximizes  the  natural 
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logeirithm  of  the  likelihood  function. 

Under  the  assumption  of  a  linear  system  with  known  system  matrices  (^,Gj,Qd,H 
and  R),  the  residuals  are  zero-mean,  white,  random  variables.  An  approximation  of  the 
residual  sample  variance  equals  53n=i  ’”n>  filter-computed  residual  variance  is 

HP.H^  -f-  Rf  Thus,  crl  is  approximately  the  ratio  of  the  sample  residual  variance  to 
the  filter-computed  residual  variance. 

Harvey  [33:107,126]  continues  by  substituting  the  ratio  of  variances  for  crj  in  the  like¬ 
lihood  equation: 

ln(i,)  =  ln(2»)  -  a  In  -  f  ln(HP.-H^  +  H.)  -  f 

=  -f(ln(2ir)  +  1)-  ^ln(HP:H^  H.)''  -  fin  {j^ZLirl)  -  f  ln(HP:H^  +  R,) 

=  -f(ln(2^)  +  l)-?ln(iEiLir-^) 

This  likelihood  equation  can  be  maximized  to  determine  parameter  estimates.  Alternatively, 

n=l 

can  be  minimized.  In  contrast  to  the  MLE  parameters,  the  least  squares  estimates  are  deter¬ 
mined  by  selecting  parameters  that  minimize  the  sum  of  the  residuals  squared,  Ef=i  ’’n- 
general,  MLE  and  least  squares  estimates  are  asymptotically  equivalent  estimates  [33:129]. 
Since  the  natural  logarithm  is  a  monotonically  increasing  function,  in  this  particular  appli¬ 
cation,  the  MLE  parameter  estimates  are  equal  to  the  least  squares  parameter  estimates  for 
fixed  sample  size  N.  However,  in  this  application  of  trying  to  separate  the  transient  and  the 
steady-state,  both  estimation  techniques,  MLE  and  least  squares,  beg  the  question  of  how 
much  of  the  data  (the  size  of  N)  to  use  in  estimating  the  steady-state  parameters. 

Harvey  [33:126]  recommends  setting  cr^  equal  to  i?  or  a  diagonal  element  in  Qd-  If 
(tI  =  R,  then  R,  =  1.0  and  Qd*  becomes  the  dynamics-to- measurement  noise  ratio.  Using 
the  modified  likelihood  function,  one  less  parameter  is  estimated,  and  the  indeterminacy 
between  Qd  and  R  should  be  eliminated.  Either  MLE  or  least  squares  estimation  can  be 
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used  to  estimate  the  remaining  parameters. 

The  scaled  vajiances  can  possibly  be  used  in  normalized  Kalman  filters  in  the  MMAE 
technique.  The  conditional  probability  of  the  jth.  filter  in  the  bank  of  K  filters  having  the 
correct  parameters  is  given  by  Equation  (36)  on  page  69.  For  the  case  of  scalar  measurements, 
substituting  in  the  normal  distribution  probability  density  function  for  the  residuals  with 

T 

mean  zero  and  variance  of  HP“H  -f  R  results  in 


Pj(tn)  = 


5Zk=l  •^2(«n)|a,Z(t„_i  )-p*(tn-l  ) 

I,-., 


T 

^  2(HP,i  H^’+r^)  J 


■Pkitn-l) 


Substituting  in  the  scaled  variances  and  al^  for  each  filter  into  the  MMAE  probability  cal¬ 
culations, 


P*j{tn) 


Using  Harvey’s  suggestion  of  setting  a^-  =  R,  so  R,j  =  1.0,  results  in  the  following  probability 
calculations: 


The  alj  do  not  cancel  and  are  unknown.  The  approach  of  assuming  the  a^j  cancel  and  setting 
an  arbitrary  value  for  each  R^j  may  result  in  a  nonlinear  change  in  the  MMAE  probability 
calculations. 
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In  a  test  of  MMAE  with  the  scaled  variances,  the  (f)i  and  Qd  were  varied  at  discrete 
levels  for  each  of  K  filters.  The  MMAE  probability  calculations  used 

Of  the  three  discrete  values  (0.1,  0.5,  and  0.9)  for  (f)\,  the  center  value  of  0.5  was  used  to 
generate  the  AR(1)  data.  After  processing  1,000  observations  through  the  Kalman  fih-'rs,  the 
marginal  probabilities  for  (j>\  were  examined.  For  each  run  with  different  .alues  of  the 
marginal  probability  for  one  of  the  discrete  values  of  <f>i  was  over  98  percent.  This  estimate 
of  <f>i  constitutes  the  Majcimum  A  Posteriori  (MAP)  estimate.  However,  different  values 
of  resulted  in  different  MAP  estimates  of  ^i.  With  =  1.0,  (^i  was  overestimated 
as  0.9.  On  the  other  hand,  when  R,j  =  0.1,  was  underestimated  as  0.1.  In  neither 
case  did  MMAE  closely  estimate  the  correct  value  of  0.5.  In  addition,  although  the 
actual  filter  residuals  were  almost  identical  for  each  value  of  R,j,  the  filter-computed  residual 
variance,  HP^H^  -f-  R,,  varied  significantly.  In  order  to  draw  reliable  statistical  inferences 
about  the  residuals,  the  filter-computed  residual  variance  should  be  approximately  equal  to 
the  actual  residual  variance.  The  choice  of  this  example  was  unfortunate  since  the  correct 
value  for  R  for  pure  AR(1)  data  without  measurement  noise  is  zero.  Therefore,  the  MMAE 
was  attempting  to  select  a  very  large  Kalman  filter  gain  proportional  to  the  ratio  of 
However,  the  approach  of  arbitrarily  fixing  Qj  or  R  and  using  MMAE  to  estimate  the 
other  parameters  does  not  appear  to  be  a  reliable  estimation  strategy.  Because  of  potential 
estimation  problems  and  unreliable  filter-computed  variance  estimates,  MMAE  is  not  applied 
in  the  algorithm  developed  in  this  chapter. 

In  an  extension  of  Harvey’s  approach,  the  original  (unstarred)  parameters  can  be  esti¬ 
mated  after  determining  estimates  of  the  starred  (*)  parameters.  These  calculations  require 
estimating  cr^  with  the  mean  squared  residual,  ^  En=i  ’’ni  ^.nd  the  filter-computed  resid¬ 
ual  variance,  HP^H^  -|-  R,.  'ine  advantage  of  this  method  is  that  the  steady-state  filter 
variance  would  be  tuned  [57:337-338]  to  the  realized  residual  variance.  Tuning  prevents  di- 
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vergence  caused  by  the  filter  relying  too  heavily  on  its  internal  dynamic  model,  and  tuning 
also  prevents  chasing  noise  in  the  data.  Tuning  the  filter  also  ensures  that  the  probability 
calculations  for  the  residuals  are  reasonable. 

Since  Harvey’s  suggested  approach  scales  the  variances  and  MMAE  probably  should 
not  be  applied,  a  simpler  formulation  is  to  let  =  HP“H^  +  R,  the  filter-computed 
residual  variance.  This  eliminates  the  need  to  calculate  the  variances  Qj,  R,  P“  or  P'*’. 
The  unknown  parameters  become  the  steady-state  mean  ,Uy,  a  scalar  k  related  to  the  Kalman 
filter  gain  K,  and  the  autoregressive  coefEcient(s)  <f>i  (and  (^2)  in  The  observations  can 
be  processed  through  Kalman  filters  with  estimates  of  these  unknown  parameters  without 
corresponding  estimates  for  Qj,  R,  P~  or  P"*".  With  only  an  estimate  of  the  Kalman  filter 
gain,  but  no  variance  estimates,  MMAE  and  MLE  can  not  be  applied,  however,  least  squares 
estimation  remains  feasible. 

In  this  chapter,  the  least  squares  estimates  for  the  steady-state  mean  fiy,  the  autore¬ 
gressive  coefficients  (f)i  and  (f>2,  and  the  scalar  k  are  used.  These  parameters  fix  the  state 
transition  matrix  $  and  the  Kalman  filter  gain  K.  Based  on  these  parameter  estimates 
and  the  resulting  residual  sequence,  the  variances  can  be  estimated  as  shown  in  the  next 
subsection. 

4-3. S  Variance  Estimates.  The  variance  relationships  for  both  the  AR(1)  with  mea¬ 
surement  noise  and  the  AR(2)  with  measurement  noise  models  are  developed.  Using  esti¬ 
mates  for  the  output  mean,  autoregressive  coefficients,  and  the  first  element  of  the  Kalman 
filter  gain,  the  simulation  output  sequence  is  processed  through  the  Kalman  filter.  The 
resulting  Kalman  filter  residuals  along  with  the  parameter  estimates  are  used  to  calculate 
variance  estimates.  If  desired,  the  unknown  variances  of  R  and  Qj,  and  thus  P"  and  P'*',  are 
calculated  using  the  mean  squared  residuals.  Further,  oased  on  the  parameter  estimates  and 
the  estimates  of  Qd  and  R,  the  simulation  output  variance  and  the  mean  estimator  variance 
are  determined. 

In  the  AR(1)  with  measurement  noise  formulation,  H  =  Gj  =  1,  K  =  A:  and  ^  =  4>i. 
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Using  the  estimates  for  ^iy,4>\  and  fc,  the  data  is  processed  through  the  Kalman  filter.  The 
approximate  residual  setmple  variance  is  set  equal  to  the  filter-computed  residual  variance: 


■'''  n=l 

Using  the  Equation  (30)  for  the  Kalman  gain, 


(48) 


After  multiplying  and  substituting,  this  relationship  results  in 

By  substituting  into  the  residual  variance  equation,  R  is  determined: 

'■"-‘'(ir-) 

Furthermore,  the  Kalman  filter  equations  with  the  estimate  for  P~  are  used  to  solve  for  P"*" 
and  Qd-  From  Equation  (31), 

P+  =  P-  -  kp- 


Using  Equation  (29), 

Qd  =  P-- 

In  addition,  since  k  =  and  both  P~  and  R  are  variances,  the  scalar  k  is  bounded 

between  zero  and  one.  For  an  AR(1)  with  measurement  noise  formulation,  the  noise  variances 
and  the  state  estimate  variances  are  determined  from  the  parameter  estimates  and  the  mean 
of  the  squared  residuals  with  these  relationships. 

For  the  AR(2)  with  measurement  noise  model,  the  Kalman  filter  gain  is  a  vector  with 
two  elements.  After  setting  the  first  element  equal  to  the  scalar  k,  the  second  element  must 
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be  solved  in  terms  of  the  estimated  parameters.  Using  the  matrices  from  Equation  (45)  in 
Equation  (30), 


+  R 


=  ■Pu+« 

p- 

-  ■Pu+'R 


The  first  element  of  the  Kalman  filter  gain  matrix  is  set  equal  to  the  scalar  k,  so  k  = 


As  in  the  AR(1)  case,  the  parameter  k  is  bounded;  0  <  A:  <  1.  However,  to  determine 
the  second  element  in  the  Kalman  filter  gain  matrix,  — -  must  be  solved  in  terms  of  the 
estimated  parameters  (fiy,  k,  <f>i,  and  ^2)-  Pn  is  determined  using  the  Kalman  filter  equations 
along  with  the  symmetry  of  covariance  matrices.  From  the  covariance  update.  Equation  (31), 


=  P-  -  KHP- 

=  P-  -  P-H^[HP-H^  +  R]-^HP 

=  P- -  (^) 

=  p-  _  [  (Pn?  PnPu  ' 

\Pi1+R/  p-p-  (T>-\2 


P nPi2 


p~  _  (Pii p~  _  PiiPia 

11  P'+H  -^12  P-+R 

p-  _  PgPu  p-  _  (P12 

<*10  i-»—  .  «  <*00 


Pu+R 


Pu+R 


From  the  covariance  matrix  propagation  equation.  Equation  (29), 


[  ^1  ^2  1  [  Pn  ~ 


1  0 


Pi";  - 

Pfi+R 

Pfi+R 

Po";  - 

Pu+R 

•‘22 

Pu+R 

<l>i  1 
<j>2  0 


Qd  0 
0  0 


After  multiplying, 


p-  —  j.  p-  ^liPnY  .  ,  p-  ^2^11^12 

Pu  -  ■#.P„  -  +  hPn  - 
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From  the  equation  for  the  first  element  of  the  Kalman  filter  gain  matrix,  +  R). 

Substituting  for  Px\  in  the  numerator  terms, 

Pf3  =  <i>^k{P-^  +  P)  -  4>,k\P{^  +  P)  +  <t>2P{2  -  4>2kP:2 


and  solving  for  Pn 


Further, 


■^12(1  -  4>2  +  ^2k)  =  i(i>ik  -  (f>ik^){P^^  +  P) 


^2  = 


{(f)xk  -  4>ik^) 
(1  —  ^2  +  ^2^) 


(Al  +  R) 


P12  _  -  4>ik^ 

P{[  +  R  1  -<f>2  +  <i>2k 


Since  this  is  the  second  element  of  the  Kalman  filter  gain,  the  complete  gain  matrix  is 


k 

K  = 

With  these  relationships,  the  least  squares  estimates  are  found.  After  the  least  squares 
estimates  of  fiy,  k,  ^1,  and  ^2  are  determined,  using  the  relationships  above,  unique  estimates 
of  P~,  ?■*■,  Qd,  and  P  can  be  calculated,  if  desired.  From  the  estimation  of  /i,  k,  (f>i,  and  ^2,  a 
residual  sequence  exists.  Theoretically,  the  residuals  are  normally  distributed  with  mean  zero 
and  variance  HP“H^  +  P.  An  approximation  of  the  residual  sample  variance  is 
Equating  these  variances  results  in 

=HP-h’'  +  «  =  Pf,  +  iJ  (52) 

n=l 

From  Equations  (49),  (51)  and  (52),  and  Pu  are  determined.  Again  using  Equation  (52), 
P  is  estimated.  Finally,  using  Equation  (50)  and  the  fact  that  P22  —  PiiJ  Qd  is  calculated. 

In  many  simulation  applications,  an  estimate  of  the  variance  of  the  process  is  useful. 
From  fitting  these  Kalman  filter  equations,  an  estimate  of  the  variance  of  the  simulation 
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output  and  an  estimate  of  the  variance  of  the  simulation  output  mean  estimator  fiy 
can  be  determined. 

The  variance  of  the  output  is  derived  using  Equations  (34),  (46)  and  (47): 


Vn  =  /iy  + 

=  fly  +  Hx(t„)  +  u(t„)  (53) 

=  My  +  ^(^n)  +  u(<n) 

The  measurement  noise  t/(<„)  is  assumed  independent  of  the  state  estimates,  which  are  the 
autoregressive  noises  The  independence  along  with  the  facts  that  E{i;^(t„)}  :=  R  and 

that  E{Tj^{in)}  =  O’?  are  used  in  the  derivations. 

For  the  AR(1)  with  measurement  noise  formulation,  the  Tj(tn)  are  modeled  as  a  zero 
mean  AR(1)  process,  as  shown  in  Equation  (42).  Thus,  o'?  =  E[fj^{tn)]  =  (j?^)  [11:58]  of 


^{[Mv  +  HQ  +  v(in)]*}  -  My 


E{rj\Q}  +  E{v^iQ} 

(t?^) 


If  the  simulation  output  y„  is  from  an  AR(1)  process,  no  “lack  of  fit”  is  necessary,  and 
R  =  0.  As  expected,  with  R  =  0,  the  variance  is  the  same  as  for  an  AR(1)  process. 

For  the  AR(2)  with  measurement  noise  formulation,  the  correlated  noise  terms  Tj{Q 
are  distributed  as  a  pure  AR(2)  process.  Equation  (43),  with  process  variance  [11:62]: 


=  (ifl)  ((TT 


_Qi _ 

hy  -  ci>l 


Since  the  result  E{y„}  =  fXy  still  holds,  only  the  last  part  of  the  derivation  that  uses  R{y^} 
changes: 

=  EmQ}  +  R 
=  (i^)f _ 2- _ W/J 
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Once  again,  if  the  actual  process  is  an  AR(2),  then  R  =  0  and  the  variance  of  y(t„)  degen¬ 
erates  to  the  pure  AR(2)  process  vaxiance. 

In  discrete-event  simulations,  the  objective  often  is  to  calculate  an  estimate  of  the 
mean  output  /iy.  The  variance  of  this  estimate  is  necessary  to  form  a  confidence  interval  for 
the  mean. 

By  independence  of  the  state  estimates  x(t„)  =  rj{tn)  and  the  measurement  noise  u(i„), 
the  variance  of  their  sum  is  the  sum  of  their  variances. 

=  Vm{  i  E"  .  ^((.)}  +  E"  .  Var{»(t.)} 

=  Var{iE^=.-?(in)}+ff 

Substituting  Fishman’s  [23:247-252]  approximation.  Equation  (11)  with  p  =  2,  for  the  vari¬ 
ance  of  the  mean  estimator  of  an  AR(2)  sequence, 

0-2  _ Qi _ ^  E 

^  N{1  hY  N 

with  degrees  of  freedom  d  from ’Equation  (12)  minus  1  for  the  estimation  of  R, 

4  -1-  4<^2 

Substituting  in  the  parameter  estimates  results  in  an  estimate  of  the  output  variance 
This  variance  estimate  is  used  for  constructing  confidence  intervals  on  the  mean  estimate 
[37,  39]. 

The  variance  estimates  are  derived  from  the  estimate  of  the  scalar  k.  Empirical  tests 
shown  in  Table  53  on  page  168  indicate  that  the  scalar  k  is  overestimated.  The  bias  in 
estimating  k  also  biases  the  variance  estimates  Qd  and  R.  Furthermore,  estimates  based  on 
Qd  and  R,  such  as  dy  and  d^,  may  also  be  biased. 
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4-3.4  Application  of  Estimation  Procedure.  Two  major  issues  affect  the  application 
of  the  estimation  procedure  developed  in  the  last  subsection.  The  first  issue  is  in  which 
direction  should  the  data  be  processed,  forward  or  backwards.  In  this  chapter,  the  data  is 
processed  backwards  beginning  at  the  end  of  the  simulation  output,  however,  in  subsequent 
chapters  the  data  is  processed  foward  from  the  beginning  to  end.  The  second  issue  is  how 
many  of  the  observations  should  be  used  to  estimate  the  steady-state  system  model.  All  of 
the  data  is  not  used  since  the  transient  data  may  bias  the  parameter  estimates  necessary  for 
the  algorithm  to  detect  the  transient.  A  bank  of  filters  is  employed  to  simultaneously  evaluate 
the  feasible  range  of  parameters  until  adequate  parameters  are  accepted.  The  discretization 
for  the  parameters  upon  which  the  filters  are  based  is  also  discussed.  Various  stopping  rules 
are  considered  to  determine  when  the  estimates  from  one  of  the  filters  in  the  bank  should 
be  accepted  as  the  steady-state  system  model. 

Processing  the  data  from  the  end  of  the  simulation  output  toward  the  beginning  min¬ 
imizes  the  chance  of  biasing  the  estimated  steady-state  model  with  transient  observations. 
Another  advantage  of  estimating  the  steady-state  model  by  processing  the  data  from  the  end 
to  the  beginning  is  that  after  the  model  is  estimated,  the  model’s  residuals  may  be  tracked 
as  a  potential  indication  of  the  onset  of  significant  transient  effects. 

In  applying  the  least  squares  estimation  technique,  the  decision  must  be  made  at 
which  observation  to  stop  the  system  parameter  estimation.  Processing  the  output  backward 
from  the  simulation  termination,  in  the  same  fashion  as  Kelton  and  Law  [44]  propose,  the 
steady-state  output  is  encountered  before  the  transient  output.  While  more  steady-state 
observations  should  improve  the  confidence  in  the  estimated  parameters,  including  transient 
output  may  bias  the  system  parameter  estimates.  But  at  this  stage  in  the  algorithm,  no 
estimate  of  where  the  transient  phase  effectively  ends  is  available.  Therefore,  at  a  point 
sufficiently  early  in  the  steady-state  observations  to  prevent  including  any  transient  data, 
the  parameter  estimation  procedure  should  be  stopped. 

The  least  squares  estimates  are  generally  found  by  applying  Kalman  filters  with  various 
parameters  to  a  fixed  sample.  A  search  is  conducted  until  the  parameters  which  minimize 
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the  sum  of  squared  residuals  is  determined.  In  this  application,  the  number  of  observa¬ 
tions  to  include  in  the  sample  for  the  estimation  algorithm  is  not  obvious.  The  decision 
of  how  many  observations  to  include  in  the  parameter  estimation  phase  is  solved  partially 
by  using  a  bank  of  filters  similar  to  the  MMAE  approach.  The  bank  of  filters  are  set  up 
in  a  grid  covering  the  feasible  and  stable  parameter  space.  The  bank  of  filters  processes 
the  observations  backwards,  beginning  with  the  last  observation.  Using  the  bank  of  filters, 
various  combinations  of  parameters  are  evaluated  after  including  each  additional  observation 
to  the  estimation  sample.  As  observations  are  processed,  the  cumulative  sum  of  residuals 
squared  is  maintained  for  each  filter.  After  sufficient  observations  are  processed,  one  filter 
may  consistently  minimize  the  cumulative  sum  of  squared  residuals.  This  filter’s  parameters 
are  selected  as  the  least  squares  estimates.  As  soon  as  the  best  combinations  of  parameter 
estimates  is  apparent,  no  more  data  points  are  included  in  the  estimation  sample.  Rather 
than  using  a  predetermined  sample  size,  this  approach  applies  an  incrementally  increasing 
sample  size  until  the  estimates  are  determined.  With  the  appropriate  stopping  rule,  this 
bank  of  filters  approach  avoids  having  to  determine  which  observations  constitute  steady 
state  before  beginning  the  estimation. 

While  optimized  discretization  schemes  exist  [94,  61],  these  schemes  require  a  “truth 
model”  which  is  not  available  in  these  applications.  An  equally  spaced  discretization  is 
employed.  The  grid  of  filters  covers  the  parameter  space  with  a  filter  running  at  each 
combination  of  the  discrete  parameters.  For  each  filter  in  the  bank,  the  sum  of  its  residuals 
squared  is  maintained.  After  processing  an  observation,  the  least  square  estimates  to  that 
point  are  the  parameters  of  the  filter  with  the  minimum  sum  of  squared  residuals. 

For  initial  tests,  the  Kalman  filter  formulation  is  an  AR(1)  with  measurement  noise 
model,  and  the  tes^  data  sequence  is  generated  with  a  pure  AR(1)  model.  The  filter  bank 
includes  filters  with  the  scalar  k  discretized  from  0.0  to  1.0  at  every  0.1,  <j)\  discretized 
from  -1.0  to  1.0  at  every  0.1.  The  true  mean  is  used  in  the  filters.  After  processing  1000 
observations,  the  minimum  is  always  very  close  to  the  parameters  used  to  generate  the  data. 
However,  the  residual  sum  of  squares  is  relatively  insensitive  to  changes  in  the  scalar  k.  In 
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Table  6.  Estimation  Stopping  Criteria 


Residuitl  Properties 

Hypothesis  Test 

Zero  Mean 

Mean  Hypothesis  Test 

Known  Covariance 

Variance  Test 

Whiteness 

(Uncorrelated  in  Time) 

Potmcinteau  Test 

Durbin- Watson  Statistic 

Normally  Distributed 

Goodness  of  Fit  Test 

addition,  most  of  the  neighboring  filters  have  residual  sum  of  squares  just  a  small  percentage 
(<  2%)  lairger  than  the  minimum.  In  other  words,  the  residual  sum  of  squcires  appears 
to  be  a  relatively  flat  surface.  Therefore,  parameter  estimates  that  differ  by  less  than  0.1 
may  not  be  distinguishable  without  an  excessive  number  of  observations.  As  a  result  of  this 
conclusion,  the  parameters  are  estimated  only  to  within  0.1  in  this  chapter.  This  coarse 
discretization  enables  the  bank  of  Kalman  filters  to  span  the  discretized  feasible  parameter 
space  completely. 

After  determining  the  discretization  of  the  unknown  parameters,  a  criterion  to  accept 
the  estimates  from  the  filter  with  the  minimum  cumulative  sum  of  squared  residuals  is 
necessary.  One  approach  for  determining  a  stopping  rule  is  periodically  to  estimate  the 
parameters  and  to  test  the  fit  of  the  model.  Mehra  and  Peschon  [64]  present  three  types 
of  statistical  hypothesis  tests  for  the  residuals;  test  of  zero  mean,  test  of  covariance,  and 
test  of  whiteness.  The  assumption  of  normally  distributed  residuals  can  also  be  tested  using 
a  goodness  of  fit  test.  These  and  other  possible  techniques  to  determine  the  estimation 
stopping  rule,  shown  in  Table  6,  are  discussed.  Because  of  the  difficulty  of  implementing 
sequential  statistical  hypothesis  tests,  a  heuristic  approach  is  selected. 

Two  variants  of  the  test  for  zero  mean  are  examined.  The  hypothesis  test  to  determine 
if  the  residuals  are  statistically  zero-mean  uses  the  sample  mean,  f  =  ^  En=i  ’'n-  The  sample 
mean  is  distributed  normally  with  zero  mean  and  variance  of  -t-  72).  Therefore, 


98 


the  null  hypothesis  of  zero  mean  is  rejected  whenever 


1.96[HP-H^  +  i?] 


at  the  5  percent  confidence  level. 

While  the  previous  test  is  based  on  the  filter-computed  residual  vaxiance  HP'H^  -|-  R, 
Mehra  and  Peschon  [64]  recommend  using  the  sample  variance.  Let  the  standardized  residu¬ 
als  be  f„  =  (HP“H^-|-iE)“  Jr„,  and  the  sample  variance  equal  d?  =  {N — 
Mehra  and  Peschon  suggest  a  test  with  the  following  statistic; 


|r|  > 


which  is  the  “uniformly  most  powerful  among  all  the  tests  for  zero  mean  which  are  invariant 
with  respect  to  scaling  (or  covariance)”  [64]. 

The  hypothesis  test  evaluates  whether  the  residual  variance  equals  the  filter-computed 
variance.  Since  the  estimation  methodology  forces  these  two  variances  to  be  equal,  this  test 
is  not  appropriate  to  determine  if  the  parameter  estimates  are  sufficient.  However,  this  test 
is  discussed  later  as  a  viable  test  for  the  onset  of  the  transient  data. 

Whiteness  is  defined  as  independence  in  time.  The  test  of  whiteness  is  based  on 
estimating  the  sample  autocorrelations  c^  at  lag  i  with 


-  ^  -  r)(r„_.  -  f) 

E^Lx(rn-r-)^ 


These  sample  autocorrelations  cein  be  compared  with  their  standard  errors,  which  are  ap¬ 
proximated  usually  by  N~^ .  The  difficulty  with  this  approach  is  that  for  a  few  lags  z,  the 
true  standard  errors  may  be  much  smaller  than  the  approximation  [2:261]. 

Another  test  of  the  sample  autocorrelations  is  the  modified  Portmanteau  test  for  an 
ARMA(p,g)  time  series  model  [2:263].  In  this  test  with  N  observations  and  Kq  sample 
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autocorrelations,  the  test  statistic  Q*  is  given  by 

<3-  =  Ar(JV  +  2)2^ 

The  statistic  Q*  is  distributed  with  Kq  —p  —  q  degrees  of  freedom  where  p  +  9  is  the  total 
number  of  parameters  in  the  model. 

While  the  Portmanteau  test  evaluates  Kq  autocorrelations,  the  Durbin- Watson  statis¬ 
tic  only  tests  if  the  first  autocorrelation  coefficient  is  near  zero.  The  Durbin- Watson  statistic, 

J)  -  ^n=2(^«  ~ 

r2 

2jn=l 

is  approximately  equal  to  2(1  — p,(l)),  where  Pr(l)  is  the  first  order  autocorrelation  coefficient 
of  the  residuals.  Since  theoretically  Pr(l)  should  be  zero,  D  should  be  close  to  2.  In  addition, 
a  good  heuristic  test  for  whiteness  is  to  adjust  for  the  mean  and  then  count  the  number  of 
zero  crossing. 

A  different  approach  is  to  apply  a  goodness-of-fit  test  to  evaluate  the  cumulative  distri¬ 
bution  of  the  residuals.  Several  goodness-of-fit  tests  [96],  such  as  the  Kolmogorov- Smirnov 
test,  can  be  used  to  test  whether  the  residuals  are  distributed  normally  with  a  mean  of 
zero  and  a  variance  equal  to  the  filter-computed  variance.  The  residuals  for  the  test  can  be 
obtained  by  processing  a  different  set  of  observations  than  the  set  of  observations  used  to 
estimate  the  parameters. 

If  the  residuals  from  the  estimated  model  failed  any  of  the  hypothesis  tests,  more 
observations  can  be  included  in  the  the  least  squares  estimation.  However,  a  difficulty 
exists  with  repeatedly  applying  a  statistical  hypothesis  test  to  determine  if  the  estimated 
model  is  adequate.  Due  to  its  statistical  nature,  an  incorrect  model  may  eventually  be 
accepted  erroneously.  The  probability  of  failing  to  reject  an  inadequate  model  depends  on 
the  power  of  the  hypothesis  test,  but  all  statistical  tests  have  some  probability  of  accepting 
an  incorrect  model.  Therefore,  for  any  given  model,  the  test  can  be  applied  repeatedly  to 
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determine  if  the  acceptance  rate  is  commensurate  with  the  confidence  level  of  the  statistical 
test.  However,  repeatedly  applying  a  statistical  test  and  monitoring  the  rate  of  acceptance 
is  not  implemented  easily.  Therefore,  sequential  hypothesis  tests  are  not  applied. 

Since  applying  a  statistical  hypothesis  test  to  determine  whether  the  parameter  esti¬ 
mates  are  adequate  is  cumbersome,  a  simple  heuristic  approach  is  developed.  The  hypothesis 
tests  are  examined  for  values  that  can  be  easily  monitored  and  give  a  relatively  good  indica¬ 
tion  of  the  quality  of  the  parameter  estimates.  The  assumption  of  zero-mean  residuals  can 
be  tested  by  monitoring  the  cumulative  sum  of  the  residuals.  If  the  residuals  are  zero  mean 
and  normally  distributed,  as  assumed  in  the  proposed  system  model,  the  cumulative  sum  of 
the  residuals  should  fluctuate  stochastically  around  zero  [70:282].  Therefore,  the  parameters 
of  the  Alter  with  the  smallest  sum  of  residuals  squared  are  accepted  only  if  the  absolute  value 
of  the  cumulative  sum  of  residuals  is  within  a  small  tolerance  of  zero. 

Similarly,  the  Durbin- Watson  statistic  can  be  updated  easily  with  each  new  obser¬ 
vation.  Thus,  as  an  indication  that  the  residuals  are  white,  the  parameter  estimates  are 
accepted  only  if  the  associated  Durbin- Watson  statistic  is  above  1.9  and  below  2.1. 

These  stopping  rules  indicate  that  the  residuals  appear  to  follow  the  assumptions  of 
being  zero-mean  and  white.  In  addition,  these  rules  are  monitored  easily  as  the  observa¬ 
tions  are  processed,  and  they  do  not  require  calculating  the  Kalman  filter  variance  matrices. 
Therefore,  the  parameters  of  the  Kalman  filter  with  the  minimum  sum  of  squared  residuals 
are  tested  after  each  additional  observation  is  processed.  When  both  tests  are  passed  si¬ 
multaneously,  that  filter’s  parameters  are  selected  as  the  least  squares  estimates.  The  least 
squares  estimation  with  these  stopping  rules  completes  the  steady-state  system  identifica¬ 
tion.  After  several  tests  runs,  the  Durbin-Watson  criterion  was  discarded.  If  the  model  used 
to  generate  the  data  does  not  match  the  Kalman  filter  formulation,  the  residuals  continually 
fail  the  Durbin-Watson  criterion. 

The  selected  stopping  rule  is  based  on  monitoring  the  cumulative  sum  of  the  residuals. 
The  cumulative  sum  should  be  normally  distributed  with  mean  zero  and  variance  propor- 
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tional  to  the  sample  size.  The  simulation  output  is  processed  backwards  from  the  simulation 
end  to  the  beginning.  After  processing  {yN,yN-i,  ■  ■  •  for  J  =  0, 1, 2, . . .,  the  number 

of  generated  residuals  is  J  + 1.  The  parameters  of  the  filter  with  the  smallest  sum  of  residuals 
squared  are  accepted  only  if  the  absolute  value  of  the  cumulative  sum  of  residuals  is  below 
a  small  tolerance  times  y/J  +  1: 

j 

^  O.Iv/jTT  (58) 

i=o 

In  summary,  a  bank  of  filters,  each  with  different  parameter  estimates,  processes  the 
data  from  the  end  of  the  simulation  output  backwards  toward  the  beginning.  From  the  bank 
of  filters,  the  parameters  of  the  Kalman  filter  with  the  minimum  sum  of  squared  residuals 
are  tested  after  processing  each  additional  observation.  When  the  cumulative  sum  is  less 
than  a  small  tolerance,  those  filter’s  parameters  are  selected  as  the  least  squares  estimates. 

In  this  section,  a  procedure  to  estimate  the  system  parameters  is  developed.  The 
procedure  processes  the  data  from  the  end  of  the  simulation  output  until  the  adequate 
parameter  estimates  are  obtained.  Based  on  this  estimated  steady-state  model,  the  Kalman 
filter  can  continue  processing  the  observations  backwards  to  detect  when  transient  data  is 
encountered.  The  next  section  developes  the  algorithm  for  detecting  transient  data  and 
selecting  an  appropriate  truncation  point. 

4-4  Truncation  Point  Selection 

Once  the  steady-state  system  identification  is  accomplished,  determination  of  the  point 
where  the  transient  phase  effectively  ends  may  be  addressed.  The  concept  is  to  truncate  the 
initial  transient  data  in  order  to  obtain  a  better  estimate  of  the  steady-state  simulation 
output  mean.  Potentially,  the  appropriate  truncation  point  may  be  identified  in  several  dif¬ 
ferent  ways.  This  chapter  tests  an  approach  based  on  monitoring  the  Kalman  filter  residuals. 
When  the  simulation  output  is  processed  through  the  estimated  Kalman  filter  from  the  end 
to  the  beginning,  the  transient  effects  may  be  apparent  by  the  onset  of  a  high  occurrence  of 
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Table  7.  Truncation  Point  Selection  Criteria 


Residual  Properties 

Hypothesis  Test 

Zero  Mean 

Mean  Hypothesis  Test 

Known  Covariance 

Variance  Test 

Whiteness 

(Uncorrelated  in  Time) 

Durbin- Watson  Statistic 
Potmanteau  Test 

Normally  Distributed 

Goodness  of  Fit  Test 

All  Properties 

Joint  Probabilities 

Log  of  Conditional  Densities 
Cumulati\re-Sum  Process  Control 

unlikely  observations  through  residual  monitoring. 

Unlikely,  and  hence  potentially  transient,  observations  may  be  determined  in  a  variety 
of  ways.  Since  the  residuals  in  the  steady-state  model  are  white  and  distributed  normally  with 
a  mean  of  zero  and  the  filter-computed  vaxiance,  any  violation  of  these  assumptions  would 
indicate  that  transient  observations  are  being  processed  through  the  steady-state  Kalman 
filter.  Therefore,  any  of  the  statistical  tests  which  are  described  previously  as  a  method 
to  accept  the  estimated  model,  can  also  be  used  to  identify  transient  data.  Those  tests 
include  the  test  of  the  residual  mean,  the  tests  of  the  residual  whiteness,  and  the  goodness- 
of-fit  tests.  In  addition,  some  different  approaches  are  apparent.  Table  7  summarizes  the 
truncation  point  selection  criteria  considered.  The  approaches  that  are  not  presented  in  the 
previous  section  are  discussed.  The  cumulative-sum  (cusum)  process  control  algorithm  is 
selected  and  tested  in  this  research.  The  last  part  of  this  section  discusses  this  application 
of  the  cusum  algorithm.  The  section  concludes  with  an  example  of  the  cusum  algorithm. 

Maybeck  [57:230]  depicts  a  method  of  residual  monitoring  technique  based  on  the  sum 
of  natural  logarithms  of  con-litional  densities  of  Ni^  residuals: 


LnM  =  c{U) 


2(HP-H^  +  R) 
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The  term  c(ti)  is  a  negative  term  independent  of  the  observed  residual  values.  This  tech¬ 
nique  requires  determining  the  group  size  Nl  and  a  threshold  value  to  determine  significant 
deviations. 

A  second  alternative  is  to  apply  a  statistical  hypothesis  test  on  the  variance.  Hines  and 
Montgomery  [36:293]  describe  a  test  of  hypothesis  on  the  variance  of  a  normal  distribution. 
By  using  the  theoretical  mean  of  zero  in  calculating  the  sample  variance  of  the  residuals, 
this  hypothesis  test  also  detects  a  significant  deviation  from  zero  in  the  residual  mean.  The 
one-sided  alternative  is  described  because  if  the  transient  data  does  not  fit  the  steady-state 
model,  the  resulting  residual  variance  should  be  too  large. 

Therefore,  for  the  hypothesis  and  alternative  hypothesis  are 

Ho  :  —  o-Q 


>  crl 

Set  cr^  equal  to  the  filter- computed  residual  variance  HP“H^  +  Roi  the  steady-state  filter, 
which  is  estimated  using  the  least  squares  technique.  The  sample  variance  of  the  test  residuals 
is  approximated  with  the  mean  squared  residuals: 


52 


is  the  number  of  observations  processed  for  the  test.  The  chi-square  distributed  test 
statistic  with  —  1  degrees  of  freedom  is 


2  (Af,.  -  1)5^ 

cl 

and  the  rule  is  to  reject  Ho  if  Xo  >  Xq//  j-i- 

While  a  type  I  or  q  error  is  the  probability  of  incorrectly  rejecting  the  null  hypothesis, 
a  type  II  or  (3  error  is  the  probability  of  incorrectly  accepting  the  hypothesis  when  it  is 
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incorrect.  Hines  and  Montgomery’s  text  [36:606]  contains  operating  characteristic  curves 
which  indicate  that,  with  a  =  .05,  a  sample  size  of  =  30  is  necessary  to  have  /3  =  .1  when 
the  true  standard  deviation  is  1.5  times  the  hypothesized  standard  deviation.  Similarly,  with 
a  =  .01  a  sample  size  of  =  40  is  needed.  Therefore,  besides  determining  critical  values  of 
the  chi-square  statistic,  this  hypothesis  test  provides  insight  into  the  desired  sample  size  N^2 . 
This  technique  is  very  similar  to  Maybeck’s  since  both  use  the  sum  of  the  residuals  squared 
divided  by  the  filter-computed  variance.  However,  the  advantage  of  the  this  technique  is 
that,  in  choosing  the  probabilities  of  type  I  and  II  errors,  the  sample  size  and  critical  value 
of  the  test  statistic  can  be  determined. 

One  additional  point  about  hypothesis  tests  is  pertinent.  Even  when  the  data  is  gener¬ 
ated  from  the  hypothesized  linear  system,  the  residuals  fail  the  hypothesis  test  approximately 
lOOo!  percent  of  the  trials.  However,  during  the  transient  phase,  the  data’s  relationship  in 
time  should  be  different  than  as  predicted  by  the  steady-state  model.  Thus,  the  transient 
values  should  produce  statistical  outliers  in  the  residuals.  Therefore,  as  with  repeated  ap¬ 
plication  of  any  statisticeil  test,  the  transient  can  be  identified  by  a  sequence  of  test  results 
with  a  much  higher  failure  rate  than  the  specified  a. 

While  hypothesis  tests  have  the  problem  of  increasing  probability  of  type  I  error  for 
repeated  applications,  process  control  algorithms  can  be  designed  to  set  the  statistical  fre¬ 
quency  of  false  alarms  [70:288].  After  estimating  the  Kalman  filter  parameters  for  steady- 
state  simulation  output,  the  filter’s  residuals  may  be  tracked  to  detect  the  onset  of  transient 
data.  Of  the  many  possible  process  control  algorithms  for  monitoring  sequences  from  which 
to  choose,  the  cumulative-sum  (cusum)  quality  control  algorithm  [70:291-2;‘'j]  is  designed 
specifically  to  detect  small  changes  in  a  sequence’s  mean.  Since  the  objective  is  to  identify 
transient  simulation  observations  that  bias  the  mean  estimate,  cusum  seems  to  be  an  ap¬ 
propriate  test.  Futhermore,  the  assumptions  of  the  cusum  algorithm  are  that  the  sequence 
is  white  and  normally  distributed,  with  mean  zero  and  known  variance.  The  Kalman  filter 
residual  sequence  follows  these  assumptions. 
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The  remeiinder  of  this  section  describes  the  application  of  cusum  to  determine  ap¬ 
propriate  truncation  points  for  discrete-event  simulation  output.  A  sensitivity  analysis  is 
presented  to  determine  the  number  of  residuals  to  aggregate  in  each  cusum  observation. 
After  determining  that  each  residual  should  constitute  a  separate  cusum  observation,  the 
parameters  for  the  cusum  algorithm  are  selected.  The  section  concludes  with  an  example  of 
using  the  cusum  algorithm  to  select  a  truncation  point. 

The  cusum  approach  is  based  on  the  concept  that  the  cumulative  sum  of  residuals 
should  fluctuate  near  zero.  The  parameter  A  is  the  magnitude  of  the  shift  in  the  mean 
desired  to  be  detected,  He,  is  the  cusum  test’s  critical  value,  and  Nc,  is  the  number  of 
residuals  to  average  for  an  observation  of  the  cusum  test.  For  convenience,  the  residuals  are 
standardized  with  f„  =  [HP“H  -|-  Thus,  the  average  for  the  cusum  test  value  is 
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1 

K, 


E 
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for  2  =  1,2, .. . 


H 

Nc, 


The  cusum  test  values  are  calculated  with 


Snii)  =  max[0,  n  -  ^  +  Saii  ~  1)] 

Si{i)  =  max[0,  “  ^  -  1)] 

where  5i/(0)  =  5^(0)  =  0.  Considering  j  as  a  small  tolerance,  Sh  represents  the  cumulative 
magnitude  of  a  trend  higher  than  the  tolerance  in  the  cusum  observations.  Likewise,  Siii) 
corresponds  to  the  cumulative  magnitude  of  a  trend  lower  than  the  tolerance.  If  or 

SL{i)  exceeds  the  sequence  is  “out-of-control”  and  is  hypothesized  to  have  encountered 
transient  data.  In  addition,  let  Nh  and  Nl  be  the  number  of  consecutive  observations  for 
which  Sff(i)  and  respectively,  have  been  positive.  Then,  a  reasonable  truncation  point 

is  Nh  or  Nl,  cusum  observations  prior  to  the  failed  test  value. 

Average  Run  Length  (ARL)  is  the  expected  number  of  test  observations  processed 
before  the  cusum  test  indicates  that  the  sequence  is  “out-of-control”.  If  the  process  is  in 
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Table  8.  Cusum  Average  Run  Length  (ARL)  for  A  =  1 


Shift  in  Mean 
(Multiples  of  a=) 

He=^ 

He  =  5 

0 

168 

465 

0.25 

74.2 

139 

0.50 

26.6 

38.0 

0.75 

13.3 

17.0 

1.00 

8.38 

1.50 

4.75 

5.75 

2.00 

3.34 

2.62 

3.11 

2.19 

2.57 

4.00 

1.71 

2.01 

Note:  Table  reprinted  from  Montgomery  [70:295] 


steady  state,  the  ARL  would  indicate  the  expected  number  of  tests  performed  between 
occurrence  of  false  aJarms.  False  alarms  occur  when  the  algorithm  indicates  that  transient 
data  has  been  encountered  when  actually  only  steady-state  data  has  been  processed.  In 
general,  a  large  ARL  is  desired  when  the  steady-state  data  is  being  encountered,  and  a  short 
ARL  when  trcinsient  data  is  processed.  By  selection  of  A,  He,  and  Nc,,  the  ARL  can  be 
varied. 

The  following  discussion  examines  the  sensitivity  of  varying  the  number  of  residuals 
in  a  cusum  observation.  The  fastest  detection  of  a  small  shift  in  the  mean  is  with  only  one 
residual  in  the  cusum  observation,  Ne  =  1.  Montgomery  [70:295]  presents  Table  8  of  ARL 
values  when  He  =  4  or  He  =  5  with  A  =  1.  The  table  assumes  the  cusum  observations 
are  standard  normal  random  variables.  Therefore,  Montgomery  selects  these  values  with  the 
objective  of  detecting  a  one  standard  deviation  shift  in  the  mean  of  the  process. 

Luceis  [55]  describes  a  methodology  for  selecting  the  parameters  for  a  cusum  control 
scheme.  He  recommends  that  the  “in-control”  expected  ARL  be  about  500  cusum  observa¬ 
tions.  In  addition,  the  cusum  test  for  this  application  is  designed  to  detect  a  shift  in  mean 
of  the  standardized  residuals  of  one  half  standard  deviation,  so  A  = 
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Table  9.  Cusum  Batch  Size  Na  Effect  on  Average  Run  Length  (ARL) 


Parameters 
Ha  A  Ha 

0 

Shift  in  the  Residual  Mean  (Multiples  of  af) 

0.25  0.50  0.75  1.00  1.50 

2.00 

1 

0.5 

8.59 

499/499 

94/94 

30.9/30.9 

17.5/17.5 

12.1/12.1  7.6/7.6 

5.5/5.5 

4 

1.0 

5.07 

499/1996 

38.8/155.2 

10.5/42.0 

5.8/23.2 

4.1/16.4  2.6/10.4 

2.0/8.0 

9 

1.5 

3.55 

497/4473 

22.1/198.9 

5.5/49.5 

2.2/19.8 

16 

2.0 

2.67 

504/8064 

14.6/233.6 

3.4/54.4 

1.9/30.4 

1.4/22.4 

36 

3.0 

1.70 

492/17,712 

8.1/291.6 

1.8/64.8 

«  1/36 

Note:  Values  are  expected  number  of  cusum  observations/expected  number  of  residucils 


If  the  variance  of  the  observations  used  for  the  cusum  test  is  reduced,  the  ARL  increases. 
The  variance  can  be  reduced  significantly  by  using  the  average  of  a  batch  of  residuals  for 
each  cusum  observation  rather  than  individual  residuals.  While  fewer  cusum  observations  are 
necessary  to  detect  a  change  in  the  mean,  each  of  these  observations  contains  more  residuals. 
Lucas’  graphs  [55]  show  that  Ha  varies  approximately  linearly  with  log(ARL).  Using  this 
interpolation  scheme  and  his  table  of  values,  the  tradeoff  between  the  cusum  batch  size  Na 
and  the  ARL  can  be  seen  in  Table  9.  Lucas  provides  values  only  for  the  sample  sizes  shown. 

As  the  cusum  batch  size  increases  from  1  to  4,  the  standard  deviation  of  each 
cusum  observation  is  dtv.xC'ased  by  ^  =  |.  Thus  the  magnitude  of  o.t  with  Ncs  =  1  equals 
the  magnitude  of  2a f  with  =  4,  as  seen  by  comparing  Table  8  and  Table  9.  From 
Table  9,  the  expected  ARL  of  =  4  and  a  mean  shift  of  0.25CTf  is  38.8  cusum  observations. 
Furthermore,  0.25crf  =  0.5ct^  when  Na  =  4.  In  Table  8,  expected  ARL  for  a  shift  in  the 
mean  of  0.5crf  is  38.0  with  Ha  =  5.  The  diff^erence  between  the  two  values  can  be  explained 
by  the  slight  change  in  the  critical  value  Ha  from  5.07  to  5.0. 

Increasing  the  batch  size  for  a  cusum  observation  reduces  the  variance  of  the  cusum 
observation.  Therefore,  large  batches  can  detect  a  shift  in  the  residual  mean  in  fewer  cusum 
observations.  However,  the  advantage  of  increased  batch  size  appears  to  be  outweighed  by 
the  additional  residuals  required  in  each  cusum  observation.  In  every  case  in  Table  9,  a 
batch  size  Na  larger  than  one  requires  more  residuals  to  detect  the  same  magnitude  of  shift 
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in  the  mean.  Therefore,  for  this  application,  a  batch  size  of  one,  Nc,  =  1,  is  used. 

With  Nc3  =  1,  selection  of  the  magnitude  of  the  shift  to  detect  A  and  the  desired 
“in-control”  ARL  determines  the  cusum  critical  value  Hcs-  Lucas  [55]  recommends  that 
the  “in-control”  ARL  be  about  500  cusum  observations,  and  Gan  [26]  presents  graphs  from 
which  to  determine  the  Hca  given  A  and  the  ARL.  The  applied  cusum  test  is  designed  to 
detect  a  shift  in  mean  of  the  standardized  residuals  of  one  half  a  standard  deviation,  so 
A  =  \af  =  The  resulting  critical  value  He,  is  8.6.  The  “in-control”  ARL  is  500,  and  the 
theoretical  “out-of-control”  ARL  for  a  shift  of  A  in  the  residuals  is  about  31  observations. 
In  actual  application,  results  may  be  degraded  from  these  theoretical  values  due  to  modeling 
assumptions  and  estimation  errors. 

After  the  cusum  test  indicates  an  “out-of-control”  signal,  a  simple  test  for  a  false  alarm 
is  to  reset  Sh  and  Sl  to  zero  without  reseting  Nh  or  N^.  If  the  data  has  really  changed 
from  the  steady-state  model,  the  cusum  algorithm  should  quickly  indicate  “out-of-control” 
again.  Therefore,  if  a  second  “out-of-control”  signal  is  calculated  within  15  observations, 
the  truncation  point  is  selected.  The  observation  counters  Nh  and  Ni  are  not  reset  so  the 
selected  truncation  point  rio  remains  where  the  residuals  sequence  first  indicate  a  systematic 
change. 

Let  the  parameter  estimation  have  completed  on  the  (A  —  J)th  observation,  continue 
processing  backwards  through  the  data  {yN-j-\,yN-j-2,  ■  •  • ,  l/i}  with  the  estimated  Kalman 
filter.  For  convenience,  the  residuals  are  transformed  to  standard  normal  random  variables, 
rpf-j  =  [HP'H  -f-  R]~3ritr-j-  The  applied  cusum  test  values  are 

5/f(;)  =  max[0,  tn-j  +  -  1)]  (59) 

SlU)  =  max[0,  -  rN-j  -  Y  +  -  1)1  (60) 

where  Sh{J)  =  Sl{J)  =  0  and  j  =  J  +  1,  J  +  2, . . . ,  N 
If  the  cusum  test  for  a  high  trend  Snij)  is  above  the  critical  value  He,  then  the  cusum  test 
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Figure  10.  Simulation  Output 


for  a  low  trend  5l(;)  must  be  be  zero.  If  ShU)  >  He,  then  SL{j)  =  0  and  Nl  =  0.  The 
reverse,  if  5£,(j)  >  He,  then  SuU)  =  0  and  Nh  =  0,  is  also  true.  Therefore,  one  possible 
estimated  truncation  point  no  is  where  the  cusum  observations  began  a  trend  that  leads  to 
exceeding  the  critical  value.  This  truncation  point  can  be  ccilculated  for  both  the  high  and 
low  “out-of-control”  signals  with 

no  =  N  —  i  Nh  +  N^  for  ShU)  >  He  or  S’l(j)  >  He  (61) 

Figure  10  shows  sample  simulation  output  from  an  AR(1)  process.  The  mean  of  the 
process  has  a  step  shift  from  10  to  12  at  observation  500.  Figure  11  shows  the  corresponding 
cusum  test  statistic  Sh-  The  test  statistic  is  reset  when  it  exceeded  the  critical  value  of  8.6. 
Upon  the  second  crossing  within  15  cusum  tests,  the  truncation  point  is  estimated  at  Nh 
observations  prior  where  Snij)  first  began  the  sequence  of  consecutive  positive  values.  As 
shown  in  Figure  11,  the  selected  truncation  point  is  observation  502  when  the  actual  transient 
ended  at  the  500th  observations.  The  simulation  output  prior  to  the  truncation  point  is  not 
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Figure  11.  Cusum  Test  Statistic  ShU) 


shown. 

This  section  has  developed  a  truncation  point  selection  algorithm  based  on  residual 
monitoring.  For  the  estimated  Kalman  filter,  the  simulation  output  sequence  is  processed 
through  the  Kalman  filter  backwards.  The  residuals  are  monitored  by  the  cusum  statistical 
process  control  algorithm.  When  the  cusum  algorithm  indicates  that  sequence  is  “out-of¬ 
control”,  the  cusum  statistics  are  reset  to  test  for  a  false  alarm.  Upon  a  second  “out-of¬ 
control”  signal  within  a  small  number  of  observations,  the  algorithm  assumes  transient  data 
is  being  processed.  The  truncation  point  is  selected  where  the  cusum  statistics  started  the 
trend  leading  to  the  first  “out-of-control”  signal.  The  next  section  describes  the  specific  steps 
in  the  algorithm. 

4-5  Residual  Monitoring  Truncation- Point  Selection  Algorithm  Steps 

Many  alternative  methods  have  been  discussed  for  developing  a  truncation-point  se¬ 
lection  algorithm.  This  section  summaries  the  techniques  that  are  tested  and  details  the 
specifics  of  their  implementation. 
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This  technique  is  a  post-processing  approach,  meaning  that  the  simulation  model  is  run 
and  the  univariate  output  is  stored  in  a  file  before  beginning  the  truncation-point  algorithm. 
An  AR(1)  or  AR(2)  system  model  with  measurement  noise  is  fit  to  the  simulation  output 
with  least  squares  estimation.  Applying  the  Kalman  filter,  the  residuals  are  monitored  with 
the  cusum  quality  control  algorithm  to  identify  an  appropriate  truncation  point  tiq.  The 
steps  of  the  truncation-point  selection  algorithm  are: 


Step  1.  Using  the  last  250  of  the  simulation  observations,  {yN,yN-i,  ■  ■  ■yN-2^9},  make  an 
initial  estimate  of 

2  249 


Step  2.  Set  up  a  bank  of  Kalman  filters  with  each  filter  having  a  different  combination  of 
the  discrete  parameters,  s-nd  k.  Discretize  the  parameters  in  increments  of 

0.1.  The  mean  estimates  vary  from  -0.3  to  -1-0.3  of  fiy  estimated  in  step  1.  k  varies 
from  0.0  to  1.0.  The  autoregressive  coefficients  and  ^2  are  varied  across  the  stable 
range  in  increments  of  0.1.  Exclude  =  0  since  this  is  the  same  as  ik  =  0.  For  AR(2) 
formulation,  only  stable  combinations  of  ^1  and  ^2  [H]  are  used,  resulting  in  a  bank 
of  3,773  or  fewer  filters.  Also  for  each  AR(2)  filter,  calculate  an  associated  K  using 
Equation  (51).  The  state  estimates  x(<o  )  are  initialized  with  zeros. 

Step  3.  For  each  Kalman  filter  in  the  bank,  beginning  with  the  last  observation  process 
the  observations.  Propagate  using  Equations  (22)  and  update  with  Equations  (47), 
(26)  and  (27).  For  each  filter,  maintain  a  cumulative  sum  of  the  squared  residuals.  For 
each  observation  after  the  (fV  —  250)th,  if  the  filter  with  the  minimum  sum  of  squared 
residuals  also  meets  the  criterion  in  Equation  (58),  select  the  parameters  of  that  filter 
as  the  least  square  estimates.  If  no  parameters  are  estimated  prior  to  encountering  the 
end  of  the  simulation  data,  steady  state  is  assumed  not  to  have  been  achieved  in  that 
output  sequence. 

Optional  Step.  To  reduce  computer  run  time,  particularly  for  Monte  Carlo  tests,  after 
observations  yN-ioo,  yN-200,  ^-nd  yN-joo  processed,  turn  off  any  Kalman  filters 
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with  a  sum  of  squared  residuals  greater  than  1.5  times  the  minimum  sum  of  squared 
residuals. 

Step  4.  Assume  the  model  estimation  in  Step  2  completes  on  the  {N  —  J)th  observation. 
The  cusum  algorithm,  Equations  (59)  and  (60),  is  initiated  using  the  residuals  of  the 
least  squares  Kalman  filter.  Set  Sh{J)  =  Sl{J)  =  0  and  continue  processing  backwards 
beginning  with  the  first  obse^  j,tion  not  used  in  least  squares  estimation.  When  either 
of  the  cusum  test  values,  ShU)  or  SlU),  exceed  the  control  limit.  He,  =  8.6,  the  point 
where  the  cusum  test  value  first  became  positive,  as  calculated  with  Equation  (61),  is 
considered  a  potential  truncation  point  no-  This  point  is  tested  for  a  false  alarm  in  the 
control  algorithm  by  resetting  the  cusum  test  values,  Snij)  and  *S'x,(i),  to  zero.  If  Sff^j) 
or  ScU)  exceeds  He,  again  within  15  observations,  ho  is  selected  as  the  truncation  point. 
Otherwise,  the  cusum  algorithm  is  continued  until  a  new  potential  truncation  point 
is  determined.  If  no  truncation  point  is  selected  in  the  entire  stream,  the  algorithm 
failed. 

Optional  Step.  Determine  Q4,  R,  or  with  the  least  squares  parameter  estimates 
and  the  approximate  residual  sample  variance. 

4.6  Monte  Carlo  Results 

A  Monte  Carlo  analysis  is  presented  to  show  the  effectiveness  of  using  the  Kalman 
filter  residual  monitoring  approach  to  determine  the  appropriate  data  truncation  point.  This 
analysis  consists  of  a  loop  that  generates  a  set  of  data,  applies  the  truncation-point  selection 
algorithm,  evaluates  the  selected  point,  and  collects  the  evaluation  statistics. 

Although  there  is  not  a  widely  accepted  or  proven  analytical  approach  for  simulation 
initial  data  truncation,  Welch’s  [101]  graphical  approach  is  the  most  commonly  applied 
technique.  The  disadvantages  of  Welch’s  technique  is  that  it  requires  a  subjective  assessment, 
and  therefore,  it  is  difficult  to  test  in  a  Monte  Carlo  analysis. 
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If  the  generated  data  sets  have  a  known  transition  point  from  the  transient  phase  to  the 
steady-state  phase,  two  performance  measures  are  the  number  of  observations  between  the 
known  and  the  selected  truncation  point  and  the  associated  MSE.  Schruben’s  [87]  univariate 
test  for  presence  of  initial  bias,  described  in  Section  2.4.1,  can  also  be  used  to  determine  if 
the  truncation  point  deleted  enough  data. 

Wilson  and  Pritsker  [104]  state  that  the  true  test  for  any  technique  is  the  effect  on  our 
ability  to  draw  inferences  from  the  data.  They  recommend  forming  confidence  intervals  on 
systems  with  analytical  steady-state  values.  Then  alternative  techniques  can  be  compared 
by  their  resulting  confidence  interval  half-widths  and  their  actual  coverage  rates. 

The  specific  performance  measures  for  the  Monte  Carlo  analysis  axe  as  follows: 

1.  The  minimum,  majcimum,  bias,  and  MSE  of  the  selected  truncation  points. 

2.  The  coverage  rate  for  confidence  intervals  calculated  with  nonoverlapping  and  overlap¬ 
ping  batch  means  [69]  technique  using  Fishman’s  algorithm  [22,  23]  to  determine  batch 
size. 

3.  The  percentage  of  truncation  sequences  that  pass  Schruben’s  test  for  no  initial  bias 
(87|. 

Besides  knowing  the  theoretical  process  mean  of  the  generated  data,  the  process  variance  is 
also  known  and  is  used  to  evaluate  the  variance  estimate. 

The  data  used  to  test  the  Kalman  filters  truncation-point  selection  algorithm  is  gen¬ 
erated  with  time-series  models.  For  these  models,  the  data  is  generated  so  that  the  onset  of 
steady  state  is  known.  The  generated  transient  phases  are  caused  by  four  types  of  changes 
in  the  simulation  output  mean.  The  four  transient  types  are  a  jump  discontinuity,  a  ramp 
change,  a  decaying  exponential,  or  a  decaying  sinusoidal.  The  test  data  sets  are  generated 
from  AR(2)  and  ARMA(1,1)  model  sequences  with  an  induced  transient.  The  Monte  Carlo 
results  are  divided  into  two  corresponding  sections. 
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Table  10.  AR(2)  Parameters  for  Generated  Data 


(j>2 

/’y(l) 

Ea 

mm 

ng 

+0.83 

9.09 

11.11 

BQ 

ill 

+0.67 

2.40 

4.00 

+0.5 

+0.3 

+0.71 

2.24 

25.00 

+0.5 

-0.3 

+0.38 

1.29 

1.56 

-0.5 

+0.3 

2.24 

msm 

1 

0 

-0.3 

1.29 

■9 

-0.67 

2.40 

ig 

is 

-0.83 

9.09 

EitnM 

4.6.1  Autoregressive  Test  Cases.  A  Monte  Carlo  analysis  is  conducted,  and  both  the 
AR(1)  with  measurement  noise  and  AR(2)  with  measurement  noise  formulations  are  tested. 
For  each  combination  of  parameters  and  conditions,  1,000  pure  AR(2)  data  sequences  are 
generated.  The  transients  are  induced  by  varying  the  mean  of  the  sequence.  Four  types  of 
transients,  step  change,  ramp,  exponential  decay,  and  sinusoidal  decay,  are  used.  Each  of 
these  transients  are  tested  at  four  different  magnitudes  or  rates  of  change. 

Pure  AR(2)  data  is  generated  with  Equation  (43).  The  specific  parameters  used  to 
generate  the  AR(2)  data,  shown  in  Table  10,  are  selected  to  produce  output  with  a  wide 
range  of  autocorrelation  and  partial  autocorrelation  curves.  For  each  of  the  generated  data 
sets,  the  variance  of  the  dynamic  noise  Qj  is  1.0,  and  no  measurement  noise  is  added,  so 
R  =  0.  The  theoretical  autocorrelation  coefficient  at  lag  one,  Pv(l)  =  indicates  that 

most  of  the  selected  models  have  relatively  high  autocorrelation.  The  sequence  variances 
given  in  Equation  (54),  have  a  wide  range.  The  last  column  reports  the  sample  size  N  times 
Fishman’s  approximate  variance  of  the  estimated  mean  [23:249]  calculated  with  the  true 
parameters  in  Equation  (11). 

Transients  for  the  time  series  models  are  induced  by  changing  the  expected  value  of 
the  generated  sequence.  Four  types  of  transients,  depicted  in  Figure  12,  are  used.  The  first 
transient  type  is  a  simple  step  change  in  the  mean.  The  next  type  is  a  ramp  or  linear  change 
in  the  mean.  The  final  two  transient  types  are  an  exponential  decay  and  an  exponentially 
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Figure  12.  Types  of  Transients 


decaying  sinusoidal  in  the  mean  of  the  output.  For  each  run,  500  transient  observations  and 
1,000  steady-state  values  are  generated.  In  order  to  compare  between  transient  types,  the 
magnitude  of  the  change  in  the  expected  value  at  the  middle  of  the  transient,  observation 
250,  is  equal  for  each  of  the  transient  types.  This  shift  in  the  magnitude  of  the  expected 
value  of  the  sequence,  labeled  A/i(<7y)  in  the  tables,  is  expressed  in  multiples  of  the  standard 
deviation  of  the  generated  data  dy. 

Statistics,  shown  in  Table  11,  are  collected  on  the  technique’s  model  estimation  and 
residual  tracking.  The  results  for  both  of  the  Kalman  filter  formulations  are  shown.  The 
top  of  the  table  are  the  results  for  the  Kalman  filter  with  an  AR(1)  with  measurement  noise 
formulation,  and  the  bottom  half  of  the  table  reports  the  results  for  the  Kalman  filter  with  an 
AR(2)  with  measurement  noise  formulation.  (These  two  models  are  discussed  on  page  81.) 
The  column  ‘Fail’  indicates  the  frequency  with  which  the  least  squares  technique  failed  to 
estimate  a  steady  state  model.  Failure  to  estimate  a  model  occurred  when  the  filter  with  the 
minimum  sum  of  squared  residuals  never  met  the  test  criterion  of  near-zero  sum  of  residuals, 
Equation  (58).  When  the  AR(2)  data  is  generated  with  (f)i  of  -1.0  or  -1.5,  the  estimation 


116 


strategy  fails  to  meet  this  criterion  for  over  a  fourth  of  the  runs. 

The  next  so*  of  columns  in  Table  11  are  statistics  on  the  number  of  simulation  observa¬ 
tions  used  to  estimate  the  system  parameters.  The  last  250  observations  are  assumed  steady 
state  and  used  to  obtain  an  initial  estimate  of  the  mean,  so  the  least  squares  estimation 
is  not  permitted  to  stop  until  after  250  observations  have  been  processed.  Therefore,  the 
minimum  occurs  at  just  greater  than  250.  The  maximum  number  of  observations  used  is 
just  prior  to  1500  since  that  is  the  number  of  available  observations.  The  average  number  of 
observations  used  to  estimate  the  model  for  the  last  four  cases  is  significantly  higher  than  the 
first  four  cases.  This  may  be  because  the  last  four  cases  have  negative  first  autocorrelatioi 
coefficients. 

The  final  set  of  columns  reports  statistics  on  the  number  of  “out-of-control”  signals 
generated  by  the  cusum  algorithm  between  model  estimation  and  truncation  point  estima¬ 
tion.  After  the  model  is  estimated,  two  cusum  “out-of-control”  signals  within  15  observations 
are  necessary  to  declare  the  transient. 

The  Kalman  filter  modeled  as  an  AR(1)  with  measurement  noise  formulation  and  the 
Kalman  filter  modeled  as  an  AR(2)  with  measurement  noise  model  require  approximately 
the  same  number  of  observations  to  estimate  a  model.  In  addition,  no  difference  in  the 
number  of  cusum  alarms  is  apparent.  Since  the  generated  sequence  consisted  of  500  transient 
observations  and  1000  steady-itate  observations,  many  of  estimated  parameters  are  based  on 
some  transient  data.  In  the  last  four  cases,  which  have  negatively  correlated  sequences,  the 
estimation  algorithm  generally  requires  more  observations,  fails  to  estimate  a  steady-state 
model  at  a  higher  frequency,  and  has  fewer  cusum  alarms.  The  fewer  cusum  alarms  are 
probably  a  result  of  fewer  remaining  data  values  after  model  estimation. 

Statistics  are  also  collected  on  the  least  squares  estimated  parameters,  and 

k.  Table  12  shows  the  Kalman  filter  parameter  estimates  for  the  AR(1)  formulation.  The 
mean  estimate  fiy  is  very  near  the  steady-state  mean  of  10.0.  Since  the  data  is  actually 
AR(2),  the  value  of  <f>i  attempts  to  account  for  the  correlation  produced  by  (pi  and  <p2- 
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Table  11.  Kalman  Filter  Model  Estimation  Summary 


Data  Parameters 

Fail 

Observations  Used  To  Estimate 
Min  Majc  Avg  St  Dev 

Cusum  Alarms 

Min  Max  Avg  St  Dev 

AR(2)  Data  with  AR(1)  with  Measurement  Noise  Model 

4>i  =  ~ 
- 

<f>i  =  - 
4>i  =  - 


252 

1447 

384.7 

179.6 

1 

24 

4.7 

2.6 

252 

1347 

303.3 

70.4 

2 

21 

4.8 

2.5 

252 

1460 

411.8 

239.2 

0 

21 

5.6 

2.8 

254 

1497 

508.7 

301.9 

0 

22 

4.5 

2.4 

265 

1499 

808.0 

384.5 

0 

17 

4.0 

2.3 

272 

1499 

1057.4 

351.2 

0 

18 

3.2 

2.0 

296 

1499 

1191.0 

244.2 

0 

17 

2.8 

1.8 

305 

1499 

1197.8 

174.1 

0 

18 

3.0 

2.1 

AR(2)  Data  with  AR(2)  with  Measurement  Noise  Model 


,  =  1.5 

(O 

II 

1 

O 

bo 

0.001 

252 

1432 

300.2 

57.9 

2 

18 

6.3 

2.5 

,  =  1.0 

^2  =  -0.5 

0.003 

254 

1429 

366.2 

147.5 

2 

19 

5.6 

2.4 

=  0.5 

<f>2  —  0.3 

0.025 

252 

1496 

363.8 

183.5 

0 

22 

5.8 

2.7 

=  0.5 

to 

II 

1 

p 

0.017 

256 

1499 

555.9 

322.6 

0 

21 

4.9 

2.5 

4>i  = 

-0.5 

(f>2  —  0.3 

E 

.066 

261 

1498 

803.4 

382.5 

0 

19 

3.7 

2.1 

(t>i  = 

-0.5 

(f>2  —  — 0.3 

.210 

273 

1499 

1055.1 

349.1 

0 

19 

3.1 

1.9 

(t>i  = 

-1.0 

4>2  =  —0.5 

E 

.328 

299 

1499 

1190.4 

243.3 

0 

16 

2.5 

1.4 

<f>i  = 

-1.5 

to 

II 

1 

O 

bo 

lE 

.461 

305 

1499 

1189.2 

174.6 

0 

8 

2.1 

0.5 

represents  16,000  runs:  1,000  for  each  transient  type  and  magnitude. 
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Table  12.  AR(  1)  with  Measurement  Noise  Model  Parameter  Estimation  Summary 


Data  Parameters 


AR(2)  Data  with  AR(1)  with  Measurement  Noise  Model 


=  1.5 
=  1.0 
=  0.5 
=  0.5 
=  -0.5 
=  -0.5 
=  -1.0 
=  -1.5 


(f)2  —  —0.8 
(f)2  =  —0.5 
^2  —  0.3 
^2  =  —0.3 
<f)2  =  0.3 
<j)2  —  — 0.3 
(f>2  =  —0.5 
4>2  —  —0.8 


10.00  0.19 
10.00  0.12 
10.00  0.28 
9.99  0.08 


0.80 

0.02 

1.00 

0.00 

0.67 

0.04 

1.00 

0.00 

0.86 

0.05 

0.61 

0.08 

0.40 

0.05 

1.00 

0.00 

0.89 

0.03 

0.55 

0.11 

0.19 

0.44 

0.89 

0  29 

0.62 

0.07 

0.99 

0.05 

0.79 

0.03 

1.00 

0.00 

Note:  Each  row  represents  the  runs  that  did  not  fail  to  estimate  a  model  from  16,000  runs. 


Table  13.  AR(1)  with  Measurement  Noise  Model  Variances  Estimation  Summary 

d  "R 

Data  Parameters  cr^  _ 

AR(2)  Data  with  AR(1)  with  Measurement  Noise  Model 


=  1.5 

^2  = 

-0.8 

2.77 

0.46 

0.00 

0.00 

2.80 

0.45 

3.02 

=  1.0 

4>2  = 

-0.5 

1.33 

0.14 

0.00 

0.00 

1.57 

0.11 

1.55 

II 

o 

h  = 

0.3 

0.44 

0.11 

0.39 

0.08 

1.51 

0.41 

1.50 

4>1  =  0.5 

<t>2  = 

-0.3 

1.11 

0.09 

0.00 

0.01 

1.15 

0.07 

1.14 

4>1  =  -0.5 

4>2  — 

0.3 

0.39 

0.09 

0.55 

0.32 

1.56 

0.13 

1.50 

(f)i  =  -0.5 

<f>2  = 

-0.3 

1.02 

0.38 

0.16 

0.43 

1.19 

0.11 

1.14 

<t>x  =  -1.0 

(f>2  = 

-0.5 

1.51 

0.20 

0.03 

0.11 

1.59 

0.08 

1.55 

<f-i  =  -1-5 

4>2  = 

-0.8 

3.20 

0.55 

0.00 

0.01 

2.93 

0.20 

3.02 

Note:  Each  row  represents  the  runs  that  did  not  fail  to  estimate  a  model  from  16,000  runs. 


From  these  estimates  for  /lj,,^i,(^2>  and  k,  the  noise  variances,  Qj  and  R,  are  de¬ 
termined,  as  shown  in  Table  13.  Finally,  ay  is  calculated  by  taking  the  square  root  of 
the  variance  calculated  with  the  estimated  parameters  in  Equation  (55).  The  theoretical 
standard  deviation  for  the  generated  AR(2)  data  is  presented  for  comparison.  Where 
necessary,  the  estimates  adjusted  for  the  “lack  of  fit”  with  inflated  estimates  of  the  dynamic 
noise  variance  Qj  (greater  than  Qj  =  1.0)  and  by  adding  measurement  noise  variance  R. 
Since  ay  is  based  on  the  bias  estimates  of  Qj  and  R,  these  estimates  also  are  biased. 
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Table  14.  AR(2)  with  Measurement  Noise  Model  Parameter  Estimation  Summary 


Data  Parameters 

■ 

mm 

WSM 

■ 

mm 

Hi 

■ 

AR(2)  Data  with  AR(2)  wil 

,h  Measurement  Noise  Moc 

el 

01  =  1.5 

II 

1 

p 

bo 

10.00 

0.20 

1.54 

0.05 

-0.84 

0.04 

0.90 

0.00 

0x  =  1.0 

02  =  —0.5 

10.00 

0.11 

1.08 

0.06 

-0.58 

0.06 

0.90 

0.00 

01  =  0.5 

o 

II 

10.00 

0.28 

0.53 

0.10 

0.26 

0.08 

0.87 

0.08 

11 

o 

02  =  -0.3 

9.99 

0.08 

0.58 

0.07 

-0.35 

0.07 

0.88 

0.04 

01  =  -0.5 

to 

II 

p 

CO 

9.97 

0.07 

-0.47 

0.13 

0.35 

0.13 

0.88 

0.05 

01  =  -0.5 

02  —  — 0.3 

9.95 

0.08 

-0.62 

0.10 

-0.39 

0.09 

0.72 

0.22 

01  =  -1.0 

02  =  -0.5 

9.93 

0.08 

-1.09 

0.06 

-0.57 

0.05 

0.81 

0.12 

01  =  -1.5 

02  =  —0.8 

9.92 

0.08 

-1.53 

0.04 

-0.83 

0.04 

0.85 

0.06 

Note:  Each  row  represents  the  runs  that  did  not  fail  to  estimate  a  model  from  16,000  runs 


Table  15.  AR(2)  with  Measurement  Noise  Model  Variance  Estimation  Summary 


Data  Parameters 

- TT - 1 

Qd 

R 

■ 

m 

AR(2)  Data  with  AR(2)  with  Measurement  Noise  Model 

01  =  1.5 

02  —  —0.8 

0.70 

0.06 

0.10 

0.01 

2.91 

0.41 

3.02 

01  =  1.0 

02  =  -0.5 

0.78 

0.06 

0.10 

0.01 

1.54 

0.10 

1.55 

01  =  0.5 

02  =  0.3 

0.82 

0.13 

0.13 

0.08 

1.44 

0.23 

1.50 

01  =  0.5 

02  =  —0.3 

0.85 

0.10 

0.12 

0.04 

1.15 

0.07 

1.14 

01  =  -0.5 

02  =  0.3 

0.89 

0.14 

0.12 

0.05 

1.51 

0.13 

1.50 

01  =  -0.5 

02  =  — 0.3 

0.69 

0.23 

0.36 

0.37 

1.17 

0.06 

1.14 

01  =  -1.0 

02  =  -0.5 

0.76 

0.13 

0.28 

0.25 

1.57 

0.07 

1.55 

01  =  -1.5 

02  =  —0.8 

0.83 

0.11 

0.26 

0.18 

3.05 

0.30 

3.02 

Note:  Each  row  represents  the  runs  that  did  not  fail  to  estimate  a  model  from  16,000  runs. 


Table  14  shows  that  when  the  data  is  from  an  AR(2)  process,  the  Kalman  filter  es¬ 
timates  /tj,,  <j>i  and  02  are  very  accurate.  The  associated  variance  estimates  are  presented 
in  Table  15.  The  estimate  for  the  variance  of  the  dynamic  driving  noise  Qd  underestimates 
the  true  value  Qd  =  1.0,  but  the  measurement  noise  variance  R  is  greater  than  zero  to 
compensate. 

Generally,  the  model  estimation  is  completed  significantly  prior  to  the  transient  phase. 
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The  parameter  estimation  results  are  similar  across  the  four  transient  types.  Therefore,  the 
model  and  parameter  estimation  tables  are  summarized  across  the  four  types  of  transients. 
However,  the  results  on  estimating  the  end  of  the  transient  phase  are  very  dependent  on  the 
type  of  transient  and  are  tabulated  separately  for  each  transient. 

The  performance  measures  are  selected  to  evaluate  the  effectiveness  of  the  Kalman 
filter  residual  monitoring  approach  to  select  an  appropriate  truncation  point  rio.  Since  the 
transient  phase  ends  in  the  generated  sequence  at  the  500th  observation,  the  accuracy  is 
evaluated  in  terms  of  this  true  truncation  point  Uo-  Tables  16,  17,  18,  and  19  report  statistics 
that  compare  the  estimated  truncation  point  ho  with  the  true  truncation  point  tIq.  Each 
transient  type  and  mean  shift  magnitude  combination  is  tested  with  1,000  Monte  Carlo  runs. 

The  column  ‘Fail’  indicates  the  frequency  with  which  no  truncation  point  is  selected 
after  the  model  is  estimated.  This  type  of  failure  generally  occurs  only  in  a  small  percentage 
of  the  runs,  except  for  the  case  where  the  transient  is  a  step  change  with  the  smallest 
magnitude  of  shift,  ^Afiy{cry).  This  high  failure  rate  to  select  a  truncation  point  occurs 
because  of  a  combination  of  two  transient  characteristics.  First,  the  step  transient  is  the 
only  one  of  the  four  transient  types  for  which  the  magnitude  of  shift  did  not  continue  to 
increase  as  the  algorithm  approaches  the  beginning  of  the  simulated  output.  Second,  the 
amount  of  shift  in  the  mean,  ^ay,  is  too  small  for  the  cusum  algorithm  to  detect.  The  cusum 
algorithm  is  designed  to  detect  a  shift  of  one  half  a  standard  deviation  in  the  magnitude  of 
the  residuals  The  transient  with  the  smallest  change  is  apparently  too  small  to  detect 

in  many  output  sequences. 

Negative  truncation-point  errors  (n„  —  <  0)  indicate  that  transient  data  is  retained 

in  the  truncated  sequence,  and  positive  errors  {ho  —  rio  >  0)  result  from  excessive  truncation. 
In  other  words,  negative  truncation- point  errors  indicate  the  average  number  of  transient 
observations  that  remain  in  the  truncated  sequence.  In  contrast,  positive  truncation  errors 
show  the  number  of  steady-state  observations  that  are  truncated.  For  example,  as  shown 
under  the  column  label  “Avg”  in  Table  16,  the  average  truncation-point  error  for  the  transient 
type  “2.0  Ramp”  is  -29.3.  For  this  case,  the  average  selected  truncation-point  is  470.7 
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while  the  generated  end  of  transient  is  at  500.  For  the  sixteen  data  types  and  transient 
combinations  tested,  the  AR(1)  formulation  results  in  average  selected  truncation-points  no 
significantly  in  the  transient. 

Tables  20,  21,  22,  and  23  report  similar  truncation  error  statistics  for  the  AR(2)  with 
measurement  noise  formulation  applied  to  the  same  AR(2)  time  series  models  and  transient 
types.  Generally,  the  AR(2)  formulation  improves  the  results  when  compared  to  the  AR(1) 
results.  It  is  not  surprising  that  the  AR(2)  formulation  is  able  to  better  fit  and  detect  a  change 
in  AR(2)  data.  The  improvement  is  apparent  in  smaller  average  truncation  errors.  The 
standard  deviation  of  the  truncation  points  and  the  Mean  Square  Error  (MSE)  sometimes 
increases  and  occasionally  decreases  when  comparing  the  AR(1)  to  the  AR(2)  formulation. 

In  general,  the  truncation-point  selection  algorithm  picks  points  near  the  end  of  the 
induced  transient.  As  the  the  shift  in  the  transient  mean  decreases  from  2ay  to  only  ^cTy,  the 
selected  truncation  point  rio  includes  more  transient  data,  as  anticipated.  The  exponentially 
decaying  sinusoidal  is  the  most  difficult  transient  to  identify,  probably  since  the  transient 
varies  about  the  steady-state  mean. 

Since  the  general  objective  of  steady  state  analysis  is  to  draw  inferences  about  the  out¬ 
put  distribution,  appropriate  performance  measures  should  reflect  the  effect  of  the  estimated 
truncation  point  on  our  ability  to  construct  confidence  intervals  [44,  52,  104,  105].  Since  the 
sampling  distribution  is  typically  assumed  to  be  symmetric  about  the  point  estimator,  the 
half  width  of  the  confidence  interval,  along  with  realized  coverage,  is  generally  used  for  com¬ 
parisons.  The  effectiveness  of  confidence  interval  construction  in  terms  of  coverage  and  half 
widths  is  reported  for  the  nonoverlapping  batch  means  (NORM)  and  the  overlapping  batch 
means  (OEM)  [69]  techniques.  (See  page  26  for  description.)  Since  the  truncated  sequence 
should  be  without  initial  bias,  the  final  criterion  selected  is  the  ability  to  pass  Schruben’s 
test  for  no  initial  bias  [87].  For  comparison,  the  same  performance  measures  are  reported 
for  the  sequence  truncated  at  the  induced  true  truncation  point  no-  A  nominal  rate  of  0.9  is 
arbitarily  selected  for  each  criterion. 
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Table  16.  AR(1)  with  Measurement  Noise  Model  Truncation  Point  Errors  (no  —  rio) 


“v(o-v) 

Type 

Fail 

Min 

Max 

Avg 

St  Dev 

MSE 

AR(2 

1  Data  with 

=  1.5  and  (f>2 

=  -0.8 

2.0 

Step 

0.000 

-15 

724 

21.2 

88.1 

8211.9 

2.0 

Ramp 

0.000 

-165 

708 

-29.3 

100.4 

10934.9 

2.0 

Exp 

0.000 

-53 

717 

21.1 

90.5 

8640.2 

2.0 

Sine 

0.004 

-427 

710 

15.8 

101.9 

10640.7 

1.5 

Step 

0.000 

-43 

727 

27.6 

105.5 

11901.5 

1.5 

Ramp 

0.000 

-248 

733 

-47.0 

112.6 

14883.2 

1.5 

Exp 

0.000 

-231 

705 

16.4 

95.4 

9378.7 

1.5 

Sine 

0.004 

-437 

709 

-3.1 

97.5 

9516.1 

1.0 

Step 

0.003 

-443 

690 

22.3 

101.3 

10761.2 

1.0 

Ramp 

0.000 

-318 

732 

-85.7 

128.2 

23784.7 

1.0 

Exp 

0.000 

-306 

715 

-61.4 

117.9 

17676.0 

1.0 

Sine 

0.002 

-382 

717 

-119.7 

138.1 

33399.1 

0.5 

Step 

0.072 

-470 

739 

-52.7 

158.4 

27873.8 

0.5 

Ramp 

0.006 

-462 

700 

-190.2 

158.1 

61164.2 

0.5 

Exp 

0.000 

-392 

723 

-193.5 

148.9 

59598.7 

0.5 

Sine 

0.000 

-382 

699 

-237.9 

160.1 

82236.0 

AR(2: 

Data  with 

=  1.0  and  (^2 

=  -0.5 

2.0 

Step 

0.000 

-2 

40 

2.8 

5.5 

37.6 

2.0 

Ramp 

0.000 

-159 

22 

-78.9 

32.1 

7256.4 

2.0 

Exp 

0.000 

-148 

81 

-39.3 

35.2 

2783.0 

2.0 

Sine 

0.000 

-201 

469 

-88.2 

50.5 

10325.0 

1.5 

Step 

0.000 

-12 

45 

2.8 

5.6 

39.0 

1.5 

Ramp 

0.000 

-220 

2 

-113.4 

38.4 

14325.8 

1.5 

Exp 

0.000 

-211 

37 

-113.6 

46.6 

15082.4 

1.5 

Sine 

0.000 

-260 

-5 

-169.0 

43.5 

30457.8 

1.0 

Step 

0.000 

-222 

72 

-7.6 

24.3 

649.1 

1.0 

Ramp 

0.000 

-305 

-23 

-174.5 

47.7 

32732.0 

1.0 

Exp 

0.000 

-278 

10 

-193.3 

43.4 

39241.5 

1.0 

Sine 

0.000 

-324 

-76 

-242.7 

34.8 

60138.2 

0.5 

Step 

0.698 

-459 

578 

-143.4 

140.1 

40193.9 

0.5 

Ramp 

0.100 

-472 

-103 

-340.1 

76.1 

121447.4 

0.5 

Exp 

0.000 

-336 

-135 

-273.0 

33.6 

75686.6 

0.5 

Sine 

0.000 

-384 

-183 

-310.6 

24.9 

97091.6 

Note:  Negative  Truncation  points  indicate  transient  observations  are  retained. 
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Table  17.  AR(1)  with  Measurement  Noise  Model  Truncation  Point  Errors  (np  —  rio) 


A/i„(<Ty) 

Type 

Fail 

Min 

Max 

Avg 

St  Dev 

MSE 

AR(2)  Data  with  <f)\ 

=  0.5  and  <f)2 

=  0.3 

2.0 

Step 

0.000 

-10 

699 

27.0 

97.7 

10263.4 

2.0 

Ramp 

0.003 

-364 

748 

-18.4 

115.4 

13656.9 

2.0 

Exp 

0.000 

-388 

732 

25.8 

119.8 

15016.1 

2.0 

Sine 

0.000 

-450 

684 

-8.1 

121.3 

14778.8 

1.5 

Step 

0.000 

-21 

731 

29.8 

106.2 

12158.9 

1.5 

Ramp 

0.003 

-170 

723 

-33.6 

121.3 

15832.9 

1.5 

Exp 

0.000 

-420 

737 

-8.0 

131.7 

17400.8 

1.5 

Sine 

0.000 

-424 

725 

-61.8 

153.1 

27258.2 

1.0 

Step 

0.008 

-48 

717 

24.8 

100.2 

10657.5 

1.0 

Ramp 

0.002 

-223 

721 

-58.7 

138.5 

22611.5 

1.0 

Exp 

0.000 

-234 

744 

-79.8 

148.2 

28333.7 

1.0 

Sine 

0.000 

-454 

740 

-138.0 

162.3 

45400.2 

0.5 

Step 

0.035 

-452 

677 

-20.7 

142.9 

20841.2 

0.5 

Ramp 

0.002 

-406 

744 

-147.1 

176.7 

52857.7 

0.5 

Exp 

0.000 

-417 

738 

-167.2 

172.8 

57835.7 

0.5 

Sine 

0.000 

-460 

734 

-219.4 

172.1 

77772.8 

AR(2)  Data  with 

=  0.5  and  <^2 

=  -0.3 

2.0 

Step 

0.004 

-428 

75 

-5.7 

32.4 

1085.3 

2.0 

Ramp 

0.000 

-325 

27 

-92.2 

44.4 

10477.6 

2.0 

Exp 

0.000 

-341 

54 

-84.6 

48.4 

9503.5 

2.0 

Sine 

0.000 

-450 

7 

-145.2 

64.3 

25203.0 

1.5 

Step 

0.005 

-453 

62 

-9.0 

46.5 

2245.9 

1.5 

Ramp 

0.000 

-386 

3 

-121.7 

46.9 

16998.2 

1.5 

Exp 

0.000 

-382 

26 

-141.7 

44.8 

22075.0 

1.5 

Sine 

0.001 

-460 

568 

-201.5 

60.1 

44221.1 

1.0 

Step 

0.021 

-449 

72 

-24.2 

60.4 

4227.3 

1.0 

Ramp 

0.001 

-455 

-26 

-183.2 

56.7 

36781.9 

1.0 

Exp 

0.000 

-433 

271 

-207.6 

44.7 

45113.8 

1.0 

Sine 

0.002 

-468 

312 

-259.9 

49.9 

70035.5 

0.5 

Step 

0.710 

-457 

41 

-170.5 

131.6 

46381.0 

0.5 

Ramp 

0.095 

-471 

-74 

-342.7 

73.5 

122821.1 

0.5 

Exp 

0.000 

-458 

-85 

-274.0 

40.8 

76769.3 

0.5 

Sine 

0.010 

-474 

-214 

-319.3 

34.1 

103105.4 

Note:  Negative  Truncation  points  indicate  transient  observations  are  retained. 
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Table  18.  AR(1)  with  Measurement  Noise  Model  Truncation  Point  Errors  {hp  —  no) 


^vi^v) 

Type 

Fail 

Min 

Max 

Avg 

St  Dev 

MSE 

AR(2 

1  Data  with  <^i 

=  —0.5  and  (f>2 

=  0.3 

2.0 

Step 

0.000 

-296 

555 

-35.4 

77.9 

7327.2 

2.0 

Ramp 

0.000 

-400 

596 

-85.8 

106.3 

18660.6 

2.0 

Exp 

0.000 

-343 

524 

-53.8 

100.8 

13066.6 

2.0 

Sine 

0.008 

-485 

611 

-105.9 

149.3 

33524.9 

1.5 

Step 

0.000 

-376 

619 

-49.3 

110.0 

14532.0 

1.5 

Ramp 

0.000 

-431 

379 

-102.3 

96.1 

19705.7 

1.5 

Exp 

0.000 

-393 

605 

-85.2 

109.7 

19288.8 

1.5 

Sine 

0.005 

-484 

564 

-146.7 

151.5 

44481.8 

1.0 

Step 

0.004 

-467 

635 

-62.0 

111.9 

16368.4 

1.0 

Ramp 

0.003 

-484 

587 

-137.2 

119.4 

33064.4 

1.0 

Exp 

0.000 

-447 

639 

-133.0 

106.8 

29082.4 

1.0 

Sine 

0.005 

-487 

658 

-217.0 

143.3 

67606.7 

0.5 

Step 

0.043 

-464 

594 

-76.9 

130.6 

22989.0 

0.5 

Ramp 

0.008 

-482 

504 

-201.4 

113.0 

53349.3 

0.5 

Exp 

0.000 

-488 

624 

-225.4 

99.8 

60766.1 

0.5 

Sine 

0.003 

-487 

659 

-286.9 

111.5 

94730.5 

AR(2)  Data  with  4>i 

=  —0.5  and  <f>2 

=  -0.3 

2.0 

Step 

0.070 

-394 

61 

-50.5 

57.8 

5894.7 

2.0 

Ramp 

0.039 

-328 

16 

-155.1 

72.9 

29382.3 

2.0 

Exp 

0.016 

-314 

5 

-141.2 

71.0 

24970.2 

2.0 

Sine 

0.046 

-481 

-19 

-298.7 

124.5 

104721.8 

1.5 

Step 

0.036 

-452 

36 

-102.7 

111.0 

22867.4 

1.5 

Ramp 

0.019 

-461 

-7 

-188.4 

83.9 

42526.5 

1.5 

Exp 

0.016 

-380 

2 

-195.6 

67.3 

42787.9 

1.5 

Sine 

0.111 

-482 

-72 

-328.9 

112.3 

120818.4 

1.0 

Step 

0.171 

-470 

40 

-123.3 

119.9 

29594.1 

1.0 

Ramp 

0.006 

-478 

-51 

-252.3 

88.2 

71431.2 

1.0 

Exp 

0.027 

-452 

-25 

-249.4 

63.1 

66169.8 

1.0 

Sine 

0.186 

-483 

-129 

-349.2 

88.9 

129863.5 

0.5 

Step 

0.886 

-450 

33 

-222.3 

135.4 

67742.6 

0.5 

Ramp 

0.232 

-476 

-99 

-371.4 

66.3 

142324.7 

0.5 

Exp 

0.032 

-489 

-113 

-310.9 

53.7 

99554.0 

0.5 

Sine 

0.264 

-485 

-233 

-354.2 

59.8 

129029.3 

Note:  Negative  Truncation  points  indicate  transient  observations  are  retained. 
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Table  19.  AR(1)  with  Measurement  Noise  Model  Truncation  Point  Errors  (np  —  rio) 


Uy{ay) 

Type 

Fail 

Min 

Max 

Avg 

St  Dev 

MSE 

AR(2)  Data  with 

=  — 1.0  and  <f)2 

=  -0.5 

2.0 

Step 

0.011 

-470 

28 

-145.4 

124.3 

36588.6 

2.0 

Ramp 

0.001 

-486 

-14 

-229.0 

118.9 

66576.0 

2.0 

Exp 

0.000 

-486 

13 

-187.6 

114.5 

48295.3 

2.0 

Sine 

0.114 

-483 

4 

-281.9 

142.3 

99683.5 

1.5 

Step 

0.029 

-468 

27 

-155.7 

125.3 

39934.8 

1.5 

Ramp 

0.002 

-483 

-38 

-237.1 

101.7 

66534.9 

1.5 

Exp 

0.000 

-483 

0 

-228.5 

102.6 

62758.4 

1.5 

Sine 

0.131 

-483 

-72 

-327.6 

115.9 

120766.1 

1.0 

Step 

0.232 

-464 

18 

-149.8 

112.8 

35170.2 

1.0 

Ramp 

0.004 

-483 

-63 

-278.9 

87.5 

85432.5 

1.0 

Exp 

0.004 

-491 

-86 

-279.3 

79.1 

84258.2 

1.0 

Sine 

0.170 

-485 

-181 

-362.9 

92.9 

140314.8 

0.5 

Step 

0.968 

-437 

3 

-281.5 

114.3 

92319.4 

0.5 

Ramp 

0.252 

-470 

-154 

-392.5 

50.2 

156579.5 

0.5 

Exp 

0.009 

-494 

-146 

-344.1 

61.0 

122093.4 

0.5 

Sine 

0.188 

-485 

-246 

-384.6 

69.2 

152685.7 

AR(2)  Data  with  4>i 

=  —1.5  and  ^2 

=  -0.8 

2.0 

Step 

0.000 

-378 

3 

-148.8 

104.0 

32968.2 

2.0 

Ramp 

0.000 

-440 

-11 

-211.7 

104.6 

55736.8 

2.0 

Exp 

0.000 

-407 

4 

-158.6 

111.9 

37671.1 

2.0 

Sine 

0.008 

-466 

2 

-173.6 

136.5 

48783.4 

1.5 

Step 

0.044 

-458 

4 

-158.7 

116.5 

38753.3 

1.5 

Ramp 

0.011 

-487 

-42 

-229.2 

102.7 

63090.8 

1.5 

Exp 

0.004 

-481 

3 

-182.7 

122.5 

48398.5 

1.5 

Sine 

0.029 

-480 

-3 

-248.1 

146.6 

83036.6 

1.0 

Step 

0.189 

-459 

10 

-147.3 

109.8 

33749.1 

1.0 

Ramp 

0.013 

-486 

-120 

-264.7 

90.3 

78208.1 

1.0 

Exp 

0.010 

-490 

-93 

-250.4 

89.1 

70626.8 

1.0 

Sine 

0.121 

-457 

-186 

-338.5 

100.7 

124757.0 

0.5 

Step 

0.997 

-301 

-119 

-210.0 

91.0 

52381.0 

0.5 

Ramp 

0.071 

-472 

-223 

-389.4 

41.6 

153351.1 

0.5 

Exp 

0.033 

-491 

-161 

-329.8 

54.7 

111746.6 

0.5 

Sine 

0.130 

-480 

-311 

-394.1 

60.7 

159022.1 

Note:  Negative  Truncation  points  indicate  transient  observations  are  retained. 
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Table  20.  ARf 2)  with  Measurement  Noise  Model  Truncation  Point  Errors  ( n 


A/ij,(o-y)  Type 


AR(2 


2.0  Step 

2.0  Ramp 

2.0  Exp 

2.0  Sine 


1.0 

Step 

1.0 

Ramp 

1.0 

Exp 

1.0 

Sine 

0.5 

Step 

0.5 

Ramp 

0.5 

Exp 

0.5 

Sine 

AR(2 

2.0 

Step 

2.0 

Ramp 

2.0 

Exp 

2.0 

Sine 

1.5 

Step 

1.5 

Ramp 

1.5 

Exp 

1.5 

Sine 

1.0 

Step 

1.0 

Ramp 

1.0 

Exp 

1.0 

Sine 

0.5 

Step 

0.5 

Ramp 

0.5 

Exp 

0.5 

Sine 

0.000 

0.000 

0.000 

0.000 


0.000 

0.000 

0.000 

0.000 


0.000 

0.000 

0.000 

0.000 


-1 

742 

-49 

738 

-1 

722 

-24 

737 

-1 

732 

-60 

692 

-4 

739 

-37 

725 

-2 

735 

-89 

728 

Avg  St  Dev 


id  <f>2  =  —0.8 


101.9 

116.2 

113.2 

108.4 


MSE 


11300.6 

13815.6 
13941.5 
12625.3 


27.9 

97.7 

10317.1 

12.2 

122.6 

15177.5 

31.7 

112.7 

13718.3 

30.1 

113.3 

13730.3 

31.0 

110.9 

13257.8 

1.2 

125.4 

15724.4 

33.5 

117.5 

14938.2 

25.7 

123.6 

15926.0 

35.8 

116.6 

14868.9 

-43.6 

125.0 

17519.7 

-46.5 

144.1 

22910.9 

-92.6 

154.2 

32356.8 

AR(2)  Data  with  =  1.0  and  <l>2  = 


0.000 

0.000 

0.000 

0.000 


0.000 

0.000 

0.000 

0.000 


0.000 

0.000 

0.000 

0.000 


0.001 

0.000 

0.000 

0.000 


16.3 
-23.1 

13.3 
3.2 


15.9 

-40.3 

-4.4 

-44.1 


17.6 

-62.6 

-77.4 

-126.8 


-0.5 


67.6 

83.2 

68.2 

74.7 


69.6 
67.8 

88.7 
86.5 


81.2 

96.1 

94.4 

108.1 


80.2 

106.1 

113.6 

128.1 


4835.9 

7457.8 
4827.2 

5591.8 


5095.7 

6222.4 

7886.3 

9430.4 


6900.1 

13159.7 

14896.0 

27770.3 


6434.2 

31054.2 

44458.7 

65173.0 


Note:  Negative  Truncation  points  indicate  transient  observations  are  retained 


Table  21.  AR(2)  with  Measurement  Noise  Model  Truncation  Point 


Mv(o-y) 

Type 

Fail 

Min 

Max 

Avg 

St  Dev 

MSE 

AR(2)  Data  with  (f>] 

=  0.5  and  ^2 

=  0.3 

2.0 

Step 

0.000 

-221 

729 

35.4 

118.1 

15207.2 

2.0 

Ramp 

0.000 

-247 

732 

-3.5 

134.5 

18095.7 

2.0 

Exp 

0.000 

-317 

745 

29.9 

122.2 

15825.8 

2.0 

Sine 

0.001 

-441 

739 

11.0 

124.6 

15657.7 

1.5 

Step 

0.002 

-64 

716 

32.9 

110.4 

13265.0 

1.5 

Ramp 

0.000 

-247 

746 

-25.7 

124.5 

16159.8 

1.5 

Exp 

0.000 

-297 

737 

4.8 

137.4 

18900.9 

1.5 

Sine 

0.000 

-470 

736 

-38.1 

153.0 

24847.6 

1.0 

Step 

0.006 

-251 

735 

28.7 

108.5 

12603.3 

1.0 

Ramp 

0.000 

-299 

721 

-46.0 

140.2 

21779.3 

1.0 

Exp 

0.000 

-318 

733 

-65.5 

150.6 

26971.5 

1.0 

Sine 

0.003 

-471 

740 

-110.2 

173.1 

42107.5 

0.5 

Step 

0.024 

-458 

742 

7.0 

122.7 

15112.9 

0.5 

Ramp 

0.004 

-467 

735 

-129.4 

174.9 

47331.8 

0.5 

Exp 

0.000 

-374 

739 

-159.1 

166.9 

53170.0 

0.5 

Sine 

0.001 

-447 

741 

-203.3 

177.8 

72951.9 

AR(2 

1  Data  with  <t>\ 

=  0.5  and  ^2  = 

=  -0.3 

2.0 

Step 

0.000 

-120 

593 

0.1 

47.0 

2213.2 

2.0 

Ramp 

0.000 

-263 

596 

-61.6 

64.8 

7989.9 

2.0 

Exp 

0.000 

-277 

543 

-34.3 

65.4 

5450.6 

2.0 

Sine 

0.000 

-482 

624 

-94.0 

100.9 

19027.8 

1.5 

Step 

0.000 

-243 

600 

-3.5 

53.1 

2828.6 

1.5 

Ramp 

0.000 

-328 

437 

-81.1 

57.2 

9857.3 

1.5 

Exp 

0.000 

-294 

595 

-94.0 

59.9 

12418.7 

1.5 

Sine 

0.001 

-475 

577 

-157.2 

98.7 

34446.5 

1.0 

Step 

0.001 

-440 

647 

-11.5 

76.4 

5968.9 

1.0 

Ramp 

0.000 

-393 

603 

-125.6 

82.8 

22631.2 

1.0 

Exp 

0.000 

-368 

639 

-162.7 

68.2 

31133.6 

1.0 

Sine 

0.006 

-473 

591 

-220.8 

84.8 

55966.4 

0.5 

Step 

0.100 

-470 

637 

-83.0 

120.8 

21485.2 

0.5 

Ramp 

0.001 

-470 

636 

-234.0 

96.4 

64021.3 

0.5 

Exp 

0.000 

-417 

647 

-237.6 

71.6 

61571.1 

0.5 

Sine 

0.017 

-478 

624 

-288.3 

79.9 

89471.6 

Note:  Negative  Truncation  points  indicate  transient  observations  are  retained. 
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Table  22.  AR(2)  with  Measurement  Noise  Model  Truncation  Point  Errors  {ho  —  Uo) 


Type 

Fail 

Min 

Max 

Avg 

St  Dev 

MSE 

AR(2 

1  Data  with 

=  —0.5  and  <f>2 

=  0.3 

2.0 

Step 

0.000 

-174 

586 

-20.7 

51.9 

3122.5 

2.0 

Ramp 

0.000 

-290 

497 

-81.9 

80.2 

13154.1 

2.0 

Exp 

0.000 

-261 

164 

-48.2 

72.1 

7528.1 

2.0 

Sine 

0.003 

-482 

355 

-117.6 

141.8 

33943.3 

1.5 

Step 

0.005 

-357 

610 

-30.4 

70.0 

5832.0 

1.5 

Ramp 

0.000 

-330 

523 

-103.7 

82.8 

17609.2 

1.5 

Exp 

0.000 

-333 

567 

-81.0 

84.3 

13662.8 

1.5 

Sine 

0.004 

-484 

326 

-171.2 

138.2 

48401.6 

1.0 

Step 

0.031 

-458 

646 

-45.4 

91.6 

10457.3 

1.0 

Ramp 

0.000 

-420 

494 

-141.3 

95.0 

28991.1 

1.0 

Exp 

0.000 

-373 

639 

-148.6 

85.2 

29340.4 

1.0 

Sine 

0.006 

-482 

658 

-237.4 

123.5 

71598.3 

0.5 

Step 

0.140 

-472 

594 

-79.6 

115.5 

19678.9 

0.5 

Ramp 

0.011 

-472 

407 

-217.1 

92.7 

55729.9 

0.5 

Exp 

0.000 

-439 

288 

-240.4 

70.6 

62782.2 

0.5 

Sine 

0.008 

-486 

-74 

-311.5 

85.0 

104283.0 

AR(2)  Data  with  <^i 

=  —0.5  and  <f)2 

=  -0.3 

2.0 

Step 

0.000 

-325 

80 

-86.0 

89.9 

15482.7 

2.0 

Ramp 

0.000 

-436 

59 

-173.6 

105.7 

41293.3 

2.0 

Exp 

0.000 

-380 

62 

-144.1 

103.5 

31484.2 

2.0 

Sine 

0.050 

-485 

22 

-268.4 

151.3 

94918.7 

1.5 

Step 

0.026 

-468 

51 

-116.9 

122.3 

28613.1 

1.5 

Ramp 

0.003 

-481 

-10 

-201.9 

116.7 

54370.5 

1.5 

Exp 

0.000 

-436 

43 

-199.2 

97.7 

49217.0 

1.5 

Sine 

0.058 

-484 

-7 

-295.0 

130.9 

104164.6 

1.0 

Step 

0.097 

-471 

66 

-119.5 

125.5 

30018.6 

1.0 

Ramp 

0.008 

-480 

-18 

-239.4 

106.9 

68715.6 

1.0 

Exp 

0.000 

-456 

-39 

-248.3 

87.6 

69338.8 

1.0 

Sine 

0.045 

-485 

447 

-328.0 

110.6 

119831.2 

0.5 

Step 

0.475 

-457 

44 

-195.1 

131.4 

55317.7 

0.5 

Ramp 

0.051 

-477 

108 

-320.0 

87.4 

110053.6 

0.5 

Exp 

0.000 

-490 

-70 

-320.8 

77.9 

108992.6 

0.5 

Sine 

0.071 

-486 

-187 

-358.1 

81.1 

134784.4 

Note:  Negative  Truncation  points  indicate  transient  observations  are  retained. 
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Table  23.  AR(2)  with  Measurement  Noise  Model  Truncation  Point  Errors  (np  —  tIo) 


Type 

Fail 

Min 

Max 

Avg 

St  Dev 

MSE 

AR(2)  Data  with  <j)i 

—  — 1.0  and  <f)2 

=  -0.5 

2.0 

Step 

0.000 

-334 

56 

-123.1 

91.3 

23477.2 

2.0 

Ramp 

0.000 

-418 

-5 

-210.3 

104.2 

55083.9 

2.0 

Exp 

0.000 

-381 

32 

-169.3 

106.1 

39930.1 

2.0 

Sine 

0.124 

-482 

40 

-269.0 

159.5 

97796.9 

1.5 

Step 

0.021 

-471 

41 

-167.4 

130.9 

45163.6 

1.5 

Ramp 

0.003 

-483 

-5 

-240.4 

113.4 

70675.7 

1.5 

Exp 

0.000 

-432 

26 

-206.1 

107.0 

53938.8 

1.5 

Sine 

0.112 

-484 

0 

-312.4 

137.6 

116507.5 

1.0 

Step 

0.055 

-466 

40 

-181.8 

135.9 

51503.5 

1.0 

Ramp 

0.008 

-482 

-26 

-270.4 

106.4 

84439.8 

1.0 

Exp 

0.000 

-475 

-18 

-271.2 

97.1 

82987.0 

1.0 

Sine 

0.197 

-485 

-51 

-343.3 

117.0 

131582.3 

0.5 

Step 

0.373 

-473 

17 

-224.6 

129.8 

67326.8 

0.5 

Ramp 

0.019 

-478 

-100 

-344.0 

76.9 

124233.9 

0.5 

Exp 

0.006 

-493 

-91 

-341.6 

75.9 

122431.6 

0.5 

Sine 

0.194 

-485 

-192 

-371.1 

86.8 

145279.2 

AR(2)  Data  with  <f)i 

=  —1.5  and  (f>2 

=  -0.8 

2.0 

Step 

0.000 

-185 

28 

-64.3 

41.2 

5829.0 

2.0 

Ramp 

0.000 

-324 

19 

-163.3 

67.5 

31218.6 

2.0 

Exp 

0.000 

-254 

13 

-99.4 

58.1 

13261.1 

2.0 

Sine 

0.004 

-477 

22 

-199.8 

142.9 

60336.3 

1.5 

Step 

0.000 

-270 

35 

-100.0 

64.3 

14134.6 

1.5 

Ramp 

0.000 

-390 

1 

-200.0 

80.5 

46495.3 

1.5 

Exp 

0.000 

-351 

19 

-153.8 

90.2 

31787.4 

1.5 

Sine 

0.028 

-480 

7 

-242.8 

154.3 

82746.1 

1.0 

Step 

0.003 

-473 

19 

-188.4 

129.1 

52176.8 

1.0 

Ramp 

0.000 

-487 

-9 

-263.1 

104.3 

80098.6 

1.0 

Exp 

0.000 

-440 

7 

-209.8 

100.7 

54174.3 

1.0 

Sine 

0.143 

-483 

1 

-296.2 

155.9 

112013.6 

0.5 

Step 

0.100 

-468 

29 

-166.9 

87.5 

35530.1 

0.5 

Ramp 

0.039 

-487 

-52 

-293.3 

71.2 

91118.5 

0.5 

Exp 

0.007 

-494 

-83 

-313.7 

78.9 

104660.1 

0.5 

Sine 

0.124 

-486 

-131 

-369.1 

97.8 

145761.4 

Note:  Negative  Truncation  points  indicate  transient  observations  are  retained. 
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For  the  confidence  interval  technique  of  NOBM,  Fishman’s  algorithm  [22,  23]  to  de¬ 
termine  the  appropriate  batch  size  is  applied.  (See  discussion  or  page  27.)  In  the  following 
tables,  the  column  labeled  ‘Fail’  indicates  the  frequency  that  Fishman’s  algorithm  did  not 
find  a  batch  size  which  results  in  statistically  independent  batches.  The  OBM  technique  is 
applied  with  the  same  batch  size  as  the  nonoverlapping  batch  means  technique.  Therefore,  if 
Fishman’s  algorithm  failed,  neither  confidence  interval  technique  is  applied  to  the  sequence. 
The  number  and  size  of  the  batches  selected  by  Fishman’s  algorithm  depended  upon  the 
type  of  variation  in  the  data. 

The  confidence  interval  and  bias  evaluations  for  the  truncated  sequence  are  presented 
in  Tables  24,  25,  26  and  27  for  the  AR(1)  with  measurement  noise  model  and  Tables  28,  29, 
30  and  31  for  the  AR(2)  with  measurement  noise  formulation.  Sequences  without  a  transient 
are  generated  by  truncating  at  the  kncwn  end  of  transient,  tiq.  The  results  for  these  sequences 
are  shown  in  the  rows  are  labeled  “None”.  Nonoverlapping  and  Overlapping  Batch  Means 
produced  practically  identical  coverage  rates,  half  widths,  and  half  width  variance  for  the 
sequences  without  a  transient.  Coverage  rates  are  generally  near  the  nominal  rate  of  0.9  for 
the  sequences  without  a  transient. 

On  the  average,  the  stimated  truncation  point  Uo  includes  some  transient  data  in  the 
truncated  data  set.  Therefore,  the  actual  coverage  rate  for  the  sequences  truncated  at  the 
estimated  truncation  point  no  decreases  from  the  data  set  truncated  at  the  true  truncation 
point  no-  Even  though  the  estimated  truncation  point  for  the  exponentially  decaying  sine 
wave  transient  generally  includes  the  most  transient  values,  the  corresponding  coverage  rates 
do  not  decrease  because  high  and  low  transient  values  apparently  cancel  each  other  out. 
For  the  “Sine”  sequences,  the  confidence  interval  half  widths  also  increase  marginally  in 
magnitude  and  variance. 

Schruben’s  initial  bias  test.  Equation  (10)  on  page  29,  requires  an  estimate  of  the 
variance  of  the  sample  mean.  The  modified  version  r''  Fishman’s  formula  to  approximate 
this  variance  for  an  autoregressive  sequence.  Equation  56,  is  applied.  When  the  population 
parameters  that  generated  the  sequence  are  used  in  the  Equation  (11),  the  variance  a?  is 
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Table  24.  AR(1)  with  Measurement  Noise  Model  Truncation  Point  ho  Evaluations 


Transient 

Type 


None 

2.0 

Step 

2.0 

Ramp 

2.0 

Exp 

2.0 

Sine 

1.5 

Step 

1.5 

Ramp 

1.5 

Exp 

1.5 

Sine 

1.0 

Step 

1.0 

Ramp 

1.0 

Exp 

1.0 

Sine 

Step 

Ramp 

Exp 

Sine 


Cov 

NOBM 

Half  Width 

Cov 

OBM 

Half  Width 

lR(2)  Data  with  4>i  =  ] 

.5  and  </»2  = 

-0.8 

0.93 

0.194 

± 

0.040 

0.91 

0.197 

± 

0.082 

0.93 

0.198 

± 

0.053 

0.93 

0.200 

± 

0.050 

0.85 

0.224 

± 

0.090 

0.85 

0.225 

± 

0.071 

0.92 

0.198 

± 

0.057 

0.92 

0.200 

± 

0.052 

0.92 

0.209 

± 

0.199 

0.92 

0.211 

± 

0.202 

0.92 

0.197 

± 

0.059 

0.93 

0.200 

± 

0.056 

0.75 

0.239 

± 

0.152 

0.75 

0.238 

± 

0.124 

0.90 

0.206 

± 

0.062 

0.90 

0.207 

± 

0.068 

0.84 

0.222 

± 

0.155 

0.85 

0.227 

± 

0.162 

0.92 

0.199 

± 

0.063 

0.92 

0.201 

± 

0.064 

0.56 

0.254 

± 

0.144 

0.56 

0.255 

± 

0.148 

0.50 

0.259 

± 

0.138 

0.50 

0.259 

± 

0.122 

0.91 

0.268 

± 

0.103 

0.92 

0.278 

± 

0.112 

None 

2.0 

Step 

2.0 

Ramp 

2.0 

Exp 

2.0 

Sine 

1.5 

Step 

1.5 

Ramp 

1.5 

Exp 

1.5 

Sine 

1.0 

Step 

1.0 

Ramp 

1.0 

Exp 

1.0 

Sine 

0.5 

Step 

0.5 

Ramp 

0.5 

Exp 

0.5 

Sine 

.238  ±  0.105 
.249  ±  0.072 
.243  ±  0.059 
.240  ±  0.063 


AR(2)  Data  with  <f>i  =  1.0  and  (^2  = 


0.109  ±  0.014 


0.014 

0.041 

0.022 


0.014 

0.118 

0.137 

0.049 


0.023 

0.104 


92 

0 

70 

0 

76 

0 

83 

0 

.91 

.61 

.46 

.87 

Dm 


Bias  Test 

m)  x\^i) 


.00 

0.91 

.70 

0.24 

.95 

0.75 

.98 

0.87 

0.117  ±  0.050 


10  ± 
18  ± 
24  ± 
37  ± 


09 

32 

40 


1.00 

0.90 

1.00 

0.91 

0.56 

0.25 

0.54 

0.31 

0.59 

0.55 

WSM 

0.92 

0.14 

0.07 

0.74 

0.63 

13 

49 

37 

20 


Note:  Since  the  nominal  rates  are  0.9,  the  coverage  and  bias  test  accuracies  are 

for  1000  runs. 


0.114  ±  0.030 
0.155  ±  0.124 
0.142  ±  0.100 
0.124  ±  0.040 


0.30  0.65  0.126  ±  0.082  0.64  0.126  ±  0.078 

0.52  0.25  0.117  ±  0.055  0.26  0.119  ±  0.063 

0.07  0.35  0.112  ±  0.042  0.34  0.111  ±  0.029 

0.00  0.86  0.107  ±  0.017  0.87  0.109  ±  0.020 


m 


31 

0.2 

03 

0.0 

23 

0.0 

0.85 


i  ±0.016 
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Table  25.  AR(1) 

with  Measurement  Noise  Model  Truncation  Point  fio  Evaluations 

Transient 

NOBM 

OEM 

Bias  Test 

A/iy(o-y) 

Type 

Fail 

Cov 

Half  Width 

Cov 

Half  Width 

F{ai) 

i 

\R(2)  Data  with  (j)i  — 

0.5  and  ^2  =  0.3 

None 

0.02 

0.86 

0.240  ±  0.072 

0.89 

0.281  ±  0.122 

0.90 

0.94 

2.0 

Step 

0.03 

0.85 

0.241  ±  0.073 

0.84 

0.232  ±  0.064 

0.91 

0.94 

2.0 

Ramp 

0.05 

0.82 

0.244  ±  0.094 

0.81 

0.240  ±  0.105 

0.79 

0.76 

2.0 

Exp 

0.03 

0.83 

0.241  ±  0.078 

0.82 

0.234  ±  0.074 

0.88 

0.88 

2.0 

Sine 

0.03 

0.82 

0.261  ±  0.145 

0.82 

0.265  ±  0.166 

0.87 

0.91 

1.5 

Step 

IQ] 

0.83 

0.236  ±  0.072 

0.82 

0.229  ±  0.064 

0.95 

1.5 

Ramp 

liftH 

0.79 

0.251  ±  0.107 

0.79 

0.246  ±  0.092 

0.72 

1.5 

Exp 

0.13 

0.78 

0.253  ±  0.110 

0.77 

0.250  ±  0.104 

0.66 

0.63 

1.5 

Sine 

0.04 

0.84 

0.276  ±  0.151 

0.86 

0.283  ±  0.171 

0.74 

0.71 

1.0 

Step 

IjiuJ 

0.83 

0.240  ±  0.079 

0.82 

0.231  ±  0.070 

0.92 

0.94 

1.0 

Ramp 

0.81 

0.275  ±  0.170 

0.79 

0.263  ±  0.125 

0.55 

0.56 

1.0 

Exp 

0.14 

0.76 

0.300  ±  0.190 

0.74 

0.284  ±  0.145 

0.38 

0.40 

1.0 

Sine 

0.02 

0.86 

0.267  ±  0.129 

0.87 

0.274  ±  0.150 

0.80 

0.81 

0.5 

Step 

■WEI 

0.83 

0.270  ±  0.200 

0.82 

0.262  ±  0.177 

0.65 

0.66 

0.5 

Ramp 

0.72 

0.284  ±  0.152 

0.75 

0.292  ±  0.168 

0.32 

0.36 

0.5 

Exp 

0.06 

0.72 

0.270  ±  0.124 

0.72 

0.271  ±  0.128 

0.40 

0.39 

0.5 

Sine 

0.01 

0.84 

0.242  ±  0.089 

0.85 

0.247  ±  0.101 

_ _ _ _ _ _ _ J 

0.80 

0.78 

AR(2)  Data  with  (f>\  =  i 

1.5  and  ^2  =  —0.3 

None 

0.00 

0.92 

0.069  ±  0.004 

0.91 

0.073  ±  0.031 

0.99 

0.89 

2.0 

Step 

||b| 

0.89 

0.072  ±  0.018 

0.90 

0.073  ±  0.028 

0.88 

0.80 

2.0 

Ramp 

0.73 

0.074  ±  0.045 

0.73 

0.074  ±  0.050 

0.45 

0.27 

2.0 

Exp 

0.10 

0.61 

0.076  ±  0.049 

0.61 

0.076  ±  0.044 

0.27 

0.15 

2.0 

Sine 

0.00 

0.85 

0.075  ±  0.043 

0.88 

0.080  ±  0.052 

0.55 

0.44 

1.5 

Step 

0.89 

0.071  ±  0.014 

0.90 

0.073  ±  0.027 

0.88 

0.79 

1.5 

Ramp 

lifiCT 

0.64 

0.074  ±  0.041 

0.63 

0.073  ±  0.034 

0.34 

0.19 

1.5 

Exp 

0.05 

0.49 

0.074  ±  0.042 

0.48 

0.073  ±  0.033 

0.13 

0.06 

1.5 

Sine 

0.00 

0.83 

0.072  ±  0.034 

0.85 

0.075  ±  0.045 

0.77 

0.71 

1.0 

Step 

0.08 

0.85 

0.072  ±  0.025 

0.85 

0.072  ±  0.025 

0.63 

1.0 

Ramp 

0.06 

0.49 

0.074  ±  0.033 

0.52 

0.076  ±  0.046 

Hi 

0.09 

1.0 

Exp 

0.03 

0.41 

0.068  ±  0.021 

0.42 

0.069  ±  0.027 

wSm 

0.05 

1.0 

Sine 

0.00 

0.86 

0.068  ±  0.026 

0.88 

0.072  ±  0.041 

0.33 

0.5 

Step 

w 

0.60 

0.073  ±  0.038 

0.61 

0.072  ±  0.036 

0.25 

0.16 

0.5 

Ramp 

0.25 

0.066  ±  0.025 

0.25 

0.066  ±  0.027 

0.06 

0.02 

0.5 

Exp 

lgl 

0.55 

0.064  ±  0.014 

0.55 

0.065  ±  0.021 

0.29 

0.10 

0.5 

Sine 

0.00 

0.87 

0.065  ±  0.026 

0.88 

0.067  ±  0.031 

0.92 

0.88 

- - - - 1— - 1 _  I _ I _ J 

Note:  Since  the  nominal  rates  are  0.9,  the  coverage  and  bias  test  accuracies  are  %  ±0.016 


for  1000  runs. 
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Table  26.  AR(1) 

with  Measurement  Noise  Model  Truncation  Point  ho  Evaluations 

Transient 

NORM 

OEM 

Bias  Test 

Type 

Fail 

Cov 

Half  Width 

Cov 

Half  Width 

AR(2)  Data  with  (f>i  =  - 

-0.5  and  <f>2  =  0.3 

None 

0.00 

0.90 

0.044  ±  0.005 

0.91 

0.049  ±  0.021 

0.85 

0.88 

2.0 

Step 

IQ 

0.80 

0.053  ±  0.034 

0.84 

0.055  ±  0.053 

0.47 

0.53 

2.0 

Ramp 

IB 

0.79 

0.064  ±  0.076 

0.81 

0.067  ±  0.094 

0.38 

0.46 

2.0 

Exp 

0.81 

0.056  ±,0.072 

0.84 

0.057  ±  0.067 

0.43 

0.50 

2.0 

Sine 

0.00 

0.86 

0.069  ±  0.089 

0.91 

0.077  ±  0.092 

0.63 

0.70 

1.5 

Step 

IQ 

0.85 

0.051  ±  0.032 

0.87 

0.053  ±  0.045 

0.47 

0.50 

1.5 

Ramp 

IB 

0.79 

0.059  ±  0.056 

0.79 

0.059  ±  0.063 

0.34 

0.43 

1.5 

Exp 

0.77 

0.054  ±  0.047 

0.77 

0.054  ±  0.051 

0.31 

0.38 

1.5 

Sine 

0.00 

0.82 

0.060  ±  0.066 

0.88 

0.071  ±  0.076 

0.61 

0.65 

1.0 

Step 

0.29 

0.83 

0.050  ±  0.030 

0.86 

0.052  ±  0.036 

0.44 

0.48 

1.0 

Ramp 

0.17 

0.77 

0.052  ±  0.033 

0.76 

0.051  ±  0.035 

0.23 

0.33 

1.0 

Exp 

0.12 

0.69 

0.048  ±  0.023 

0.71 

0.048  ±  0.027 

0.21 

0.29 

1.0 

Sine 

0.00 

0.83 

0.057  ±  0.063 

0.90 

0.069  ±  0.074 

0.56 

0.63 

0.5 

Step 

0.85 

0.050  ±  0.032 

0.84 

0.050  ±  0.028 

0.45 

0.5 

Ramp 

im 

0.69 

0.046  ±  0.018 

0.70 

0.046  ±  0.020 

0.23 

0.5 

Exp 

0.12 

0.69 

0.045  ±  0.019 

0.69 

0.045  ±  0.027 

0.15 

0.25 

0.5 

Sine 

0.00 

0.86 

0.052  ±  0.051 

0.91 

0.058  ±  0.054 

0.48 

0.57 

AR(2)  Data  with  <f>\  ^  ~ 

0.5  and  (t>2  =  -0.3 

None 

0.00 

0.91 

0.032  ±  0.005 

0.89 

0.034  ±  0.014 

0.99 

0.86 

2.0 

Step 

ml 

0.58 

0.044  ±  0.019 

0.64 

0.047  ±  0.029 

■19 

0.30 

2.0 

Ramp 

UMIM 

0.49 

0.045  ±  0.033 

0.49 

0.045  ±  0.035 

mm 

0.08 

2.0 

Exp 

0.10 

0.39 

0.042  ±  0.028 

0.39 

0.042  ±  0.027 

0.10 

0.04 

2.0 

Sine 

0.00 

0.85 

0.062  ±  0.059 

0.92 

0.071  ±  0.060 

0.75 

0.46 

1.5 

Step 

ml 

0.46 

0.042  ±  0.016 

0.54 

0.045  ±  0.023 

0.28 

0.23 

1.5 

Ramp 

ng 

0.42 

0.040  ±  0.019 

0.44 

0.040  ±  0.023 

0.13 

0.06 

1.5 

Exp 

0.10 

0.30 

0.038  ±  0.014 

0.31 

0.038  ±  0.015 

0.05 

0.02 

1.5 

Sine 

0.00 

0.81 

0.057  ±  0.055 

0.93 

0.073  ±  0.064 

0.83 

0.61 

1.0 

Step 

0.33 

0.57 

0.038  ±  0.022 

0.57 

0.039  ±  0.021 

HOI 

0.20 

1.0 

Ramp 

0.18 

0.30 

0.036  ±  0.009 

0.30 

0.037  ±  0.012 

0.02 

1.0 

Exp 

0.09 

0.31 

0.32 

0.036  ±  0.014 

WnKH 

1.0 

Sine 

0.82 

0.95 

0.060  ±  0.048 

0.5 

Step 

niMj 

0.30 

0.035  ±  0.008 

0.30 

0.035  ±  0.007 

0.10 

0.5 

Ramp 

mm 

0.14 

0.033  ±  0.005 

0.15 

0.034  ±  0.006 

0.01 

0.5 

Exp 

0.07 

0.40 

0.034  ±  0.009 

0.42 

0.035  ±  0.016 

m 

0.03 

0.5 

Sine 

0.00 

0.88 

0.036  ±  0.017 

0.92 

0.039  ±  0.022 

• - - - L_ - 1 _ _ _ I _ _ _ i _  . 

Note:  Since  the  nominal  rates  are  0.9,  the  coverage  and  bias  test  accuracies  are  «  ±0.016 


for  1000  runs. 
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Table  27.  AR(1)  with  Measurement  Noise  Model  Truncation  Point  ho  Evaluations 


Transient 

NOBM 

OBM 

Bias  Test 

Afiy{ay)  Type 

Fail 

Cov 

Half  Width 

Cov 

Half  Width 

m)  x\oi) 

AR(2)  Data  with  4>\  =  — 

1.0  and  (^2  =  —0.5 

None 

0.01 

0.94 

0.025  ±  0.006  : 

0.91 

0.024  ±  0.010 

1.00  0.82 

Step 

Ramp 

Exp 

Sine 


Step 

Ramp 

Exp 

Sine 


Step 

Ramp 

Exp 

Sine 


Step 

Ramp 

Exp 

Sine 


0.043  ±  0.018 
0.054  ±  0.044 
0.043  ±  0.033 
0.068  ±  0.067 


0.044  ±  0.013 
0.049  ±  0.031 
0.046  ±  0.032 
0.058  ±  0.056 


0.038  ±  0.017 
0.040  ±  0.016 
0.035  ±  0.013 
0.044  ±  0.040 


0.034  ±  0.019 
0.030  ±  0.008 
0.033  ±  0.013 
0.039  ±  0.031 


0.042  ±  0.017 
0.048  ±  0.032 
0.042  ±  0.024 
0.071  ±  0.067 


0.045  ±  0.015 
0.049  ±  0.032 
0.044  ±  0.029 
0.065  ±  0.055 


0.039  ±  0.016 
0.046  ±  0.028 
0.038  ±  0.021 
0.054  ±  0.041 


0.034  ±  0.018 
0.030  ±  0.010 
0.035  ±  0.021 
0.039  ±  0.026 


AR(2)  Data  with  4>i  =  —1.5  and  <f>2  =  —0.8 


None 

.09 

0.98 

0.024 

± 

0.007 

0.96 

0.021 

i 

0.007 

2.0 

Step 

.61 

0.39 

0.074 

± 

0.030 

0.31 

0.071 

± 

0.032 

2.0 

Ramp 

! 

.30 

0.33 

0.070 

± 

0.061 

0.41 

0.073 

± 

0.073 

2.0 

Exp 

I 

.46 

0.31 

0.062 

± 

0.027 

0.43 

0.066 

± 

0.028 

2.0 

Sine 

[ 

.02 

0.90 

0.076 

± 

0.064 

0.94 

0.078 

± 

0.063 

1.5 

Step 

\ 

.39 

0.28 

0.068 

± 

0.024 

0.34 

0.071 

± 

0.026 

1.5 

Ramp 

.23 

0.22 

0.059 

± 

0.038 

0.30 

0.067 

± 

0.065 

1.5 

Exp 

0 

.38 

0.29 

0.055 

± 

0.023 

0.29 

0.055 

± 

0.023 

1.5 

Sine 

0 

.00 

0.97 

0.069 

± 

0.046 

0.98 

0.070 

± 

0.045 

1.0 

Step 

.48 

0.42 

0.051 

± 

0.016 

0.48 

0.053 

± 

0.018 

1.0 

Ramp 

.30 

0.15 

0.051 

± 

0.018 

0.15 

0.051 

± 

0.018 

1.0 

Exp 

0 

.29 

0.06 

0.052 

± 

0.018 

0.06 

0.051 

± 

0.015 

1.0 

Sine 

0 

.00 

0.99 

0.064 

± 

0.036 

1.00 

0.065 

± 

0.037 

0.5 

Step 

.50 

0.00 

0.045 

± 

0.000 

0.00 

0.046 

± 

0.000 

0.5 

Ramp 

0 

.14 

0.01 

0.045 

± 

0.005 

0.00 

0.045 

± 

0.004 

0.5 

Exp 

0 

.31 

0.04 

0.042 

± 

0.011 

0.05 

0.042 

± 

0.010 

0.5 

Sine 

0 

.00 

0.93 

0.045 

± 

0.027 

0.99 

0.051 

± 

0.023 

Note;  Since  the  nominal  rates  are  0.9,  the  coverage  and  bias  test  accuracies  are 

for  1000  runs. 


0.58 


.02 

.00 

.02 

.30 


.02 

.00 

.01 

.29 


0.03 

0.00 

0.00 

0.27 


0.00 

0.00 

0.00 

0.38 

±0.016 
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Table  28.  AR(2)  with  Measurement  Noise  Model  Truncation  Point  fio  Evaluations 


Treinsient 
AnyM  Type 

Fail 

Cov 

NOBM 

Half  Width 

Cov 

OEM 

Half  Width 

Bias  Test 

AR(2)  Data  with  <l>t  =  J 

.5  and  ^2  =  —0.8 

None 

0.00 

0.93 

0.194  ±  0.039 

0.93 

0.196  ±  0.036 

0.78 

0.88 

2.0 

Step 

UIJ 

0.92 

0.198  ±  0.050 

0.93 

0.200  ±  0.049 

0.83 

0.91 

2.0 

Ramp 

0.92 

0.198  ±  0.057 

0.92 

0.199  ±  0.053 

0.75 

0.86 

2.0 

Exp 

EEI 

0.91 

0.197  ±  0.055 

0.92 

0.200  ±  0.054 

0.83 

0.92 

2.0 

Sine 

0.01 

0.91 

0.198  ±  0.055 

0.91 

0.200  ±  0.052 

0.75 

0.86 

1.5 

Step 

0.01 

0.91 

0.196  ±  0.052 

0.92 

0.198  ±  0.047 

0.84 

0.92 

1.5 

Ramp 

0.00 

0.91 

0.197  ±  0.056 

0.91 

0.200  ±  0.053 

0.73 

0.84 

1.5 

Exp 

0.00 

0.90 

0.198  ±  0.057 

0.91 

0.199  ±  0.055 

0.82 

0.91 

1.5 

Sine 

0.01 

0.92 

0.198  ±  0.054 

0.92 

0.200  ±  0.052 

0.76 

0.87 

1.0 

Step 

0.00 

0.91 

0.198  ±  0.059 

0.91 

0.199  ±  0.057 

0.83 

1.0 

Ramp 

0.01 

0.91 

0.197  ±  0.055 

0.92 

0.199  ±  0.051 

0.67 

iH 

1.0 

Exp 

0.01 

0.93 

0.200  ±  0.057 

0.92 

0.201  ±  0.054 

0.82 

0.91 

1.0 

Sine 

0.00 

0.92 

0.200  ±  0.069 

0.92 

0.201  ±  0.069 

0.80 

0.90 

0.5 

Step 

0.01 

0.92 

0.199  ±  0.058 

0.92 

0.200  ±  0.055 

IQI 

0.93 

0.5 

Ramp 

0.00 

0.89 

0.196  ±  0.054 

0.89 

0.198  ±  0.048 

mm 

0.63 

0.5 

Exp 

0.00 

0.83 

0.200  ±  0.058 

0.84 

0.202  ±  0.054 

0.34 

0.46 

0.5 

Sine 

0.00 

0.91 

0.199  ±  0.059 

0.91 

0.201  ±  0.057 

0.69 

0.79 

AR(2)  Data  with  <f>i  =  ] 

.0  and  (f>2  =  —0.5 

None 

0.00 

0.92 

0.109  ±  0.014 

0.92 

0.110  ±  0.013 

0.85 

0.89 

2.0 

Step 

0.00 

0.91 

0.111  ±  0.019 

0.92 

0.111  ±  0.018 

0.88 

0.91 

2.0 

Ramp 

0.01 

0.89 

0.109  ±  0.027 

0.89 

0.110  ±  0.019 

0.70 

0.77 

2.0 

Exp 

0.00 

0.90 

0.110  ±  0.032 

0.90 

0.111  ±  0.023 

0.84 

0.90 

2.0 

Sine 

0.00 

0.89 

0.111  ±  0.026 

0.90 

0.111  ±  0.025 

0.83 

0.89 

1.5 

Step 

0.00 

0.90 

0.111  ±  0.021 

0.91 

0.111  ±  0.019 

0.88 

0.92 

1.5 

Ramp 

0.00 

0.86 

0.109  ±  0.024 

0.86 

0.109  ±  0.019 

0.62 

0.70 

1.5 

Exp 

0.00 

0.85 

0.111  ±  0.019 

0.86 

0.112  ±  0.018 

0.64 

0.71 

1.5 

Sine 

0.00 

0.86 

0.110  ±  0.017 

0.86 

0.111  ±  0.018 

0.79 

0.83 

1.0 

Step 

0.00 

0.89 

0.110  ±  0.019 

0.90 

0.111  ±  0.020 

0.88 

0.91 

1.0 

Ramp 

0.00 

0.84 

0.109  ±  0.020 

0.84 

0.109  ±  0.020 

0.48 

0.56 

1.0 

Exp 

0.00 

0.77 

0.109  ±  0.017 

0.78 

0.109  ±  0.016 

0.33 

0.43 

1.0 

Sine 

0.00 

0.89 

0.109  ±  0.026 

0.89 

0.109  ±  0.031 

0.72 

0.79 

0.5 

Step 

0.87 

0.111  ±  0.021 

ESI 

0.111  ±  0.018 

0.76 

0.81 

0.5 

Ramp 

0.75 

0.106  ±  0.020 

HI 

0.107  ±  0.019 

0.29 

0.37 

0.5 

Exp 

0  73 

0.104  ±  0.019 

EQ 

0.105  ±  0.017 

0.26 

0.35 

0.5 

Sine 

0.88 

0.88 

0.103  ±  0.021 

0.57 

0.65 

■ - - - L-- - - — i _ i _ J _ 

Note:  Since  the  nominal  rates  are  0.9,  the  coverage  and  bias  test  accuracies  are  ^  ±0.016 

for  1000  runs. 


Table  29.  AR(2) 

with  Measurement  Noise  Model  Truncation  Point  rio  Evaluations 

Transient 

NORM 

OBM 

Bias  Test 

Afiy{ay) 

Type 

Fail 

Cov 

Half  Width 

Cov 

Half  Width 

F{al) 

AR(2)  Data  with  = 

0.5  and  (^2  =  0.3 

None 

0.03 

0.86 

0.240  ±  0.071 

0.85 

0.231  ±  0.063 

0.89 

0.94 

2.0 

Step 

0.83 

0.241  ±  0.076 

0.82 

0.233  ±  0.066 

0.90 

0.94 

2.0 

Ramp 

litixl 

0.83 

0.243  ±  0.106 

0.82 

0.238  ±  0.107 

0.78 

0.82 

2.0 

Exp 

0.03 

0.83 

0.240  ±  0.100 

0.82 

0.233  ±  0.085 

0.90 

0.92 

2.0 

Sine 

0.03 

0.83 

0.248  ±  0.111 

0.81 

0.245  ±  0.123 

0.90 

0.94 

Step 

0.83 

0.238  ±  0.077 

0.82 

0.231  ±  0.077 

0.92 

0.95 

Ramp 

lifia 

0.81 

0.245  ±  0.096 

0.79 

0.240  ±  0.082 

0.72 

0.74 

Exp 

0.06 

0.77 

0.254  ±  0.148 

0.77 

0.249  ±  0.143 

0.69 

0.74 

■91 

Sine 

0.05 

0.82 

0.261  ±  0.111 

0.82 

0.264  ±  0.133 

0.79 

0.80 

1.0 

Step 

0.84 

0.240  ±  0.078 

0.82 

0.232  ±  0.070 

0.92 

0.95 

1.0 

Ramp 

lifCT 

0.81 

0.255  ±  0.127 

0.80 

0.249  ±  0.114 

0.58 

0.64 

1.0 

Exp 

0.12 

0.77 

0.277  ±  0.154 

0.76 

0.267  ±  0.126 

0.42 

0.49 

1.0 

Sine 

0.01 

0.84 

0.255  ±  0.098 

0.84 

0.257  ±  0.113 

0.82 

0.85 

0.5 

Step 

0.83 

0.247  ±  0.112 

0.81 

0.240  ±  0.099 

0.77 

0.80 

0.5 

Ramp 

0.75 

0.271  ±  0.131 

0.77 

0.271  ±  0.140 

0.34 

0.44 

0.5 

Exp 

0.03 

0.73 

0.265  ±  0.123 

0.73 

0.263  ±  0.118 

0.37 

0.44 

0.5 

Sine 

0.01 

0.84 

0.235  ±  0.073 

0.85 

0.237  ±  0.085 

0.72 

0.79 

AR(2)  Data  with  =  C 

1.5  and  <f>2  =  —0.3 

None 

0.00 

0.92 

0.069  ±  0.004 

0.92 

0.069  ±  0.004 

0.88 

0.89 

2.0 

Step 

niiia 

0.87 

0.074  ±  0.025 

0.88 

0.075  ±  0.036 

0.78 

2.0 

Ramp 

litlW 

0.84 

0.074  ±  0.048 

0.84 

0.074  ±  0.051 

0.56 

2.0 

Exp 

0.03 

0.82 

0.072  ±  0.035 

0.82 

0.072  ±  0.028 

is 

0.54 

2.0 

Sine 

0.00 

0.84 

0.074  ±  0.048 

0.86 

0.079  ±  0.060 

0.71 

0.72 

1.5 

Step 

HMII« 

0.88 

0.072  ±  0.015 

0.90 

0.074  ±  0.032 

mm 

0.78 

1.5 

Ramp 

0.01 

0.78 

0.070  ±  0.026 

0.79 

0.070  ±  0.032 

19 

0.49 

1.5 

Exp 

0.02 

0.69 

0.070  ±  0.023 

0.69 

0.070  ±  0.027 

mm 

0.27 

1.5 

Sine 

0.00 

0.87 

0.072  ±  0.043 

0.89 

0.077  ±  0.057 

0.65 

0.66 

1.0 

Step 

0.88 

0.071  ±  0.026 

0.89 

0.072  ±  0.025 

mm 

0.75 

1.0 

Ramp 

0.70 

0.068  ±  0.015 

0.71 

0.069  ±  0.023 

Is 

0.32 

1.0 

Exp 

0.67 

0.066  ±  0.007 

0.67 

0.066  ±  0.011 

0.22 

1.0 

Sine 

0.00 

0.87 

0.068  ±  0.034 

0.88 

0.072  ±  0.042 

0.64 

0.64 

0.5 

Step 

0.69 

0.069  ±  0.011 

0.68 

0.069  ±  0.011 

039 

0.42 

0.5 

Ramp 

0.56 

0.064  ±  0.010 

0.56 

0.064  ±  0.009 

0.13 

0.15 

0.5 

Exp 

Esa 

0.71 

0.064  ±  0.014 

0.71 

0.065  ±  0.020 

0.24 

0.27 

0.5 

Sine 

0.00 

0.88 

0.067  ±  0.042 

0.89 

0.068  ±  0.038 

0.72 

0.75 

- -  -I- - -  ‘ _ 1 _ I 

Note:  Since  the  nominal  rates  are  0.9,  the  coverage  and  bias  test  accuracies  are  %  ±0.016 


for  1000  runs. 
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Table  30.  AR(2) 

with  Measurement  Noise  Model  Truncation  Point  fio  Evaluations 

Transient 

NORM 

OBM 

Bias  Test 

Type 

Fail 

Cov 

Half  Width 

Cov 

Half  Width 

n^l) 

AR(2)  Data  with  <t>\  —  - 

-0.5  and  <^2  =  0.3 

None 

0.00 

0.90 

0.044  ±  0.004 

0.90 

0.044  ±  0.004 

0.90 

0.88 

2.0 

Step 

0.055  ±  0.039 

ESI 

0.53 

0.54 

2.0 

Ramp 

Ira 

0.068  ±  0.083 

0.42 

0.43 

2.0 

Exp 

0.054  ±  0.065 

0.49 

0.50 

2.0 

Sine 

0.01 

0.069  ±  0.084 

0.72 

0.71 

1.5 

Step 

0.82* 

0.052  ±  0.028 

0.86 

0.056  ±  0.049 

0.52 

0.52 

1.5 

Ramp 

ng 

0.74 

0.058  ±  0.053 

0.77 

0.060  ±  0.064 

0.34 

0.37 

1.5 

Exp 

0.22 

0.74 

0.055  ±  0.048 

0.74 

0.056  ±  0.051 

0.30 

0.31 

1.5 

Sine 

0.01 

0.82 

0.061  ±  0.065 

0.89 

0.075  ±  0.078 

0.58 

0.57 

1.0 

Step 

0.80 

0.050  ±  0.026 

0.84 

0.052  ±  0.029 

0.52 

0.52 

1.0 

Ramp 

IQ 

0.75 

0.055  ±  0.037 

0.75 

0.054  ±  0.040 

0.24 

0.26 

1.0 

Exp 

0.13 

0.64 

0.050  ±  0.027 

0.64 

0.050  ±  0.029 

0.15 

0.18 

1.0 

Sine 

0.00 

0.83 

0.057  ±  0.060 

0.91 

0.069  ±  0.069 

0.62 

0.63 

0.5 

Step 

0.83 

0.050  ±  0.032 

0.82 

0.050  ±  0.029 

mtm 

0.41 

0.5 

Ramp 

IQ 

0.65 

0.047  ±  0.021 

0.66 

0.048  ±  0.024 

0.14 

0.5 

Exp 

0.13 

0.65 

0.045  ±  0.021 

0.65 

0.046  ±  0.029 

Ha 

0.17 

0.5 

Sine 

0.00 

0.85 

0.054  ±  0.052 

0.91 

0.061  ±  0.056 

0.56 

AR(2)  Data  with  =  — 

0.5  and  <f>2  =  —0.3 

None 

0.00 

0.93 

0.032  ±  0.005 

0.93 

0.032  ±  0.005 

0.95 

0.86 

2.0 

Step 

Ba 

0.54 

0.044  ±  0.019 

0.60 

0.047  ±  0.028 

0.25 

0.24 

2.0 

Ramp 

Ira 

0.51 

0.044  ±  0.032 

0.51 

0.044  ±  0.035 

0.15 

0.14 

2.0 

Exp 

IQ 

0.49 

0.041  ±  0.029 

0.48 

0.041  ±  0.028 

0.15 

0.13 

2.0 

Sine 

0.00 

0.83 

0.059  ±  0.066 

0.90 

0.066  ±  0.067 

0.56 

0.50 

1.5 

Step 

Bra 

0.44 

0.042  ±  0.020 

0.52 

0.046  ±  0.026 

0.22 

1.5 

Ramp 

0.45 

0.039  ±  0.018 

0.47 

0.040  ±  0.023 

B 

0.12 

1.5 

Exp 

IQ 

0.39 

0.037  ±  0.013 

0.39 

0.038  ±  0.015 

0.05 

0.05 

1.5 

Sine 

0.00 

0.82 

0.053  ±  0.060 

0.92 

0.064  ±  0.064 

0.56 

0.52 

1.0 

Step 

ESI 

0.54 

0.037  ±  0.015 

0.56 

0.038  ±  0.015 

0.27 

0.26 

1.0 

Ramp 

0.40 

0.036  ±  0.011 

0.41 

0.036  ±  0.013 

0.08 

0.07 

1.0 

Exp 

IQ 

0.39 

0.035  ±  0.011 

0.40 

0.036  ±  0.013 

0.06 

0.05 

1.0 

Sine 

0.00 

0.82 

0.042  ±  0.034 

0.93 

0.053  ±  0.044 

0.53 

0.49 

0.5 

Step 

0.40 

0.034  ±  0.007 

0.40 

0.034  ±  0.005 

mmm 

0.11 

0.5 

Ramp 

0.32 

0.033  ±  0.007 

0.32 

0.033  ±  0.007 

B 

0.04 

0.5 

Exp 

0.13 

0.43 

0.035  ±  0.012 

0.44 

0.036  ±  0.018 

B 

0.07 

0.5 

Sine 

0.00 

0.90 

0.046  ±  0.047 

0.94 

0.046  ±  0.040 

0.63 

- - ; - L_ - _ 1  _ _ , 

Note:  Since  the  nominal  rates  are  0.9,  the  coverage  and  bias  test  accuracies  are  «  ±0.016 


for  1000  runs. 
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Table  31.  AR{2)  with  Measurement  Noise  Model  Truncation  Point  hg  Evaluations 


Transient 
Any{ay)  Type 


NOBM 
Half  Width 


OBM 

Half  Width 


Bias  Test 

m) 


AR(2)  Data  with  =  —1.0  and  <f>2  —  —0.5 


None 

0.01 

0.94 

0.025  ±  0.006 

0.94 

0.025  ±  0.005 

2.0 

Step 

IQ 

0.38 

0.044  ±  0.018 

0.36 

0.043  ±  0.017 

2.0 

Ramp 

im 

0.52 

0.053  ±  0.044 

0.45 

0.048  ±  0.032 

2.0 

Exp 

0.48 

0.39 

0.043  ±  0.036 

0.37 

0.041  ±  0.026 

2.0 

Sine 

0.00 

0.92 

0.067  ±  0.068  1 

0.94 

0.071  ±  0.068 

1 

Step 

IQ 

0.25 

0.044  ±  0.013 

0.29 

0.045  ±  0.015 

1.5 

Ramp 

IB 

0.40 

0.047  ±  0.030  1 

0.43 

0.048  ±  0.032 

1.5 

Exp 

wm 

0.40 

0.043  ±  0.031 

0.39 

0.042  ±  0.029 

1.5 

Sine 

0.00 

0.90 

0.059  ±  0.057 

0.97 

0.065  ±  0.057 

1.0 

Step 

0.27 

0.039  ±  0.016 

0.31 

0.040  ±  0.015 

1.0 

Ramp 

ms 

0.27 

0.039  ±  0.017 

0.35 

0.043  ±  0.027 

1.0 

Exp 

0.34 

0.28 

0.034  ±  0.013 

0.32 

0.036  ±  0.019 

1.0 

Sine 

0.00 

0.86 

0.045  ±  0.043 

0.96 

0.053  ±  0.043 

0.5 

Step 

IBI 

0.32 

0.031  ±  0.010 

0.31 

0.031  ±  0.010 

0.5 

Ramp 

0.21 

0.029  t  0.005 

0.21 

0.029  ±  0.006 

0.5 

Exp 

0.39 

0.30 

0.032  ±  0.014 

0.33 

0.035  ±  0.022 

0.5 

Sine 

0.00 

0.96 

0.040  ±  0.035 

1 

0.97 

0.040  ±  0.029 

AR(2)  Data  with  =  —1.5  and  <f>2  =  —0.8 


None 

0.10 

0.98 

0.025 

± 

0.007 

0.98 

0.025 

± 

0.005 

2.0 

Step 

IjKM 

0.36 

0.074 

± 

0.031 

0.29 

0.071 

± 

0.034 

2.0 

Ramp 

BEI 

0.34 

0.072 

± 

0.066 

0.46 

0.077 

± 

0.082 

2.0 

Exp 

0.34 

0.32 

0.059 

± 

0.024 

0.48 

0.065 

± 

0.028 

2.0 

Sine 

0.01 

0.93 

0.082 

± 

0.066 

0.95 

0.084 

± 

0.065 

■B 

I 


Step 

Ramp 

Exp 

Sine 


Step 

Ramp 

Exp 

Sine 


Step 

Ramp 

Exp 

Sine 


I 


[ill 

Ct 

lift 

#1 

iliT 


Q 

Ql 


0.31  0.066  ±  0.023 
0.24  0.060  ±  0.043 
0.36  0.053  ±  0.028 
0.97  0.067  ±  0.047 


0.40  0.052  ±  0.016  0.44  0.053  ±  0.018 

0.26  0.048  ±  0.021  0.27  0.048  ±  0.021 

0.26  0.047  ±  0.021  0.25  0.046  ±  0.018 

0.99  0.060  ±  0.042  0.99  0.061  ±  0.043 


0.011 

0.009 

0.014 

0.029 


Note;  Since  the  nominal  rates  are  0.9,  the  coverage  and  bicis  test  accuracies  are 

for  1000  runs. 


0.045  ±  0.011 
0.040  ±  0.009 
0.040  ±  0.013 
0.049  ±  0.028 


i 


0.03 

0.02 

0.04 

0.32 


.02 

.01 

.01 

.34 


.02 

.01 

.00 

.33 


s  ±0.016 
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assumed  to  be  known  exactly  and  Schruben’s  xl  statistic  in  Equation  (9)  is  tested.  The 
assumption  of  known  variance  of  the  mean  estimator  variance  is  reasonable  since  Fishman’s 
approximation  is  asymptotically  unbiased  and  large  samples  are  used.  With  the  estimated 
variance  from  Equation  (56)  and  degrees  of  freedom  d  from  Equation  (57),  Schruben’s 
statistic,  shown  in  Equation  (10),  is  used. 

The  acceptance  rate  for  Schruben’s  test  for  initial  bias  is  very  close  to  the  nominal  rate 
of  0.9  when  the  variance  of  the  mean  estimator,  calculated  with  the  true  parameters, 
is  used.  The  results  are  slightly  less  accurate  when  the  estimated  variance  of  the  mean 
estimator  is  used.  However,  the  estimated  truncation  points  generally  result  in  a  pass 
rate  in  excess  of  50  percent  for  Schruben’s  initial  bias  test. 

Schruben’s  initial  bias  identification  test  appears  to  be  more  dependent  than  the  con¬ 
fidence  interval  techniques  on  the  type  of  variation  in  the  data.  For  the  sequence  without  a 
trcinsient,  Schruben’s  technique  worked  well  with  the  “true”  variance  estimate  and  the 
test  and  also  with  the  estimated  mean  estimator  variance  d?  and  the  F  test.  Since  the  esti- 
mated  truncation  point  ho  usually  retained  some  transient  data  in  the  sequence,  Schruben’s 
initial  bias  test  appears  to  be  extremely  sensitive  to  even  a  small  number  of  transient  data 
values. 

Tables  32,  33,  34,  and  35  show  the  number  and  size  of  the  batches  of  observations  used 
for  Nonoverlapping  and  Overlapping  Batch  Means  confidence  interval  techniques.  For  the 
last  two  cases,  (^i  =  —1.0,  ^2  =  —0.5)  and  (^i  =  —1.5,  <^2  =  —0-8),  the  average  number 
of  the  batches  for  the  sequence  truncated  at  the  estimated  truncation  point  ho  significantly 
increases  in  comparison  with  the  sequence  truncated  at  the  true  point  no.  Because  of  the 
increased  variance  in  the  batches’  sample  means,  the  confidence  interval  half  widths  actually 
increase.  However,  the  batches  which  included  transient  values  bias  the  mean  estimate 
enough  to  cause  a  significant  decrease  in  the  coverage  rate.  These  two  cases  correspond  to 
the  cases  with  significant  degradation  in  actual  coverage  rates. 

While  the  MSE  appears  relatively  equal  between  the  AR(1)  and  the  AR(2)  Kalman 
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Table  32.  Fishman’s  Number  and  Size  of  Batches  for  AR(2) 


Trcinsient 

Type 


Number  of  Batches 
Am 


66.84 

34.34 

64.96 

33.45 

65.59 

32.02 

64.65 

33.97 

65.30 

32.44 

63.57 

32.02 

67.13 

33.89 

63.55 

33.15 

66.30 

35.97 

63.95 

35.04 

67.94 

34.36 

64.77 

33.43 

65.44 

34.47 

64.51 

33.82 

75.16 

39.26 

79.02 

41.00 

83.39 

40.70 

None 


Step 

Ramp 

Exp 

Sine 


Step 

Ramp 

Exp 

Sine 


Step 

Ramp 

Exp 

Sine 


Step 

Ramp 

Exp 

Sine 


AR(2)  Data  with  =  l.Oand  (^2 


with  Measurement  Noise  Model 


Size 

Ab  djB 


=  -0.8 


19.31  11.56 


19.21 

18.91 


None 

2.0 

Step 

2.0 

Ramp 

2.0 

Exp 

2.0 

Sine 

1.5 

Step 

1.5 

Ramp 

1.5 

Exp 

1.5 

Sine 

1.0 

Step 

1.0 

Ramp 

1.0 

Exp 

1.0 

Sine 

0.5 

Step 

0.5 

Ramp 

0.5 

Exp 

0.5 

Sine 

I 


168.98 


64.99 

74.30 

62.46 

167.60 


60.24 

77.78 

75.47 

89.83 


156.72 

183.77 

197.14 

214.87 


1.42 

8.13 

17.53 

8.39 


68.09 


67.84 

70.96 

70.27 

70.92 


69.09 

73.96 

73.76 

77.34 


68.21 

77.42 

79.22 

81.92 


73.45 

85.34 

84.77 

88.33 


19.23 

19.28 


19.29 

17.50 

16.98 

16.39 


=  -0.5 


7.35 


.42 

.17 

7.88 

7.51 


.86 

.47 

.38 

.22 


8.03 

7.42 

6.91 

6.69 


7.63 

7.54 

7.47 

6.86 


11.29 

11.62 


1.71 

0.46 

2.37 

1.41 


.25 
.52 
11.57 
11.83 


11.67 

9.71 

10.63 

9.64 


4.88 


5.21 

5.31 

6.67 

5.35 


5.93 

7.09 

6.13 

7.09 


6.24 
6.56 

5.24 
6.52 


7.05 

10.63 

9.51 

4.32 
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Table  33.  Fishman’s  Number  and  Size  of  Batches  for  AR(2)  with  Measurement  Noise  Model 


Transient  Number  of  Batches  Size 

Afj.y{(7y)  Type  fiM  gm  ft-B 


AR(2)  Data  with  (f>i  =  0.5  eind  (f>2  =  0.3 


None 


40.87 


2.0 

Step 

2.0 

Ramp 

2.0 

Exp 

2.0 

Sine 

1.5 

Step 

1.5 

Ramp 

1.5 

Exp 

1.5 

Sine 

1.0 

Step 

1.0 

Ramp 

1.0 

Exp 

1.0 

Sine 

0.5 

Step 

0.5 

Ramp 

0.5 

Exp 

0.5 

Sine 

AR(2) 

None 

2.0 

Step 

2.0 

Ramp 

2.0 

Exp 

2.0 

Sine 

1.5 

Step 

1.5 

Ramp 

1.5 

Exp 

1.5 

Sine 

1.0 

Step 

1.0 

Ramp 

1.0 

Exp 

1.0 

Sine 

0.5 

Step 

0.5 

Ramp 

0.5 

Exp 

0.5 

Sine 

r 


40.43 

39.04 


16.95 


6.88 

7.28 

17.19 

17.67 


31.09 

38.58 


6.95 

8.98 

18.80 

16.31 


AR(2)  Data  with  (f>i  =  0.5  and  (^2  =  —0.3 


475.92 


429.33 

472.95 


81.83 


445.72 

493.25 

485.70 

457.39 


457.92 
501.31 

539.92 
489.74 


483.53 

557.20 

568.61 

549.51 


32.26 

20.43 

133.74 

194.57 


110.34 

132.86 

108.01 

202.53 


112.11 

129.24 

130.48 

191.41 


29.60 

14.52 

28.50 

31.23 

iicKl 

BKil 

29.25 

15.47 

31.80 

17.56 

28.08 

14.16 

33.99 

18.24 

33.19 

19.58 

36.60 

17.99 

28.88 

14.47 

37.85 

23.65 

45.92 

31.55 

36.56 

19.36 

31.26 

18.11 

55.14 

41.21 

55.57 

39.66 

37.46 

20.78 

2.30 

1.50 

6.66 

15.12 

8.22 

26.71 

4.67 

16.39 

8.73 

19.15 

5.25 

12.88 

5.30 

19.44 

5.91 

20.92 

8.64 

18.48 

3.82 

5.23 

3.18 

10.96 

8.41 

18.72 

2.91 

8.12 

3.17 

9.91 

4.21 

15.54 

6.36 

15.39 
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Table  34.  Fishman’s  Number  and  Size  of  Batches  for  AR(2)  with  Measurement  Noise  Model 


Number  of  Batches 


Mm  Mb 


AR(2)  Data  with  <f>\  =  —0.5  and  ^2  =  0.3 


None 


Step 

Ramp 

Exp 

Sine 


Step 

Ramp 

Exp 

Sine 


Step 

Ramp 

Exp 

Sine 


Step 

Ramp 

Exp 

Sine 


297.99 


229.83 

211.03 

228.29 

181.07 


232.60 

212.84 

218.01 

173.49 


222.75 
212.09 
219  38 
160.39 


240.67 

222.48 

259.64 

195.64 


128.07 


57.99 

64.16 

55.45 

72.09 


159.65 

157.33 

154.06 

187.04 


156.04 


171.31 


4.14 


15.29 

31.14 

18.28 

22.61 


15.00 

28.62 

21.50 

24.28 


15.25 
30.68 
22.44 

25.25 


15.90 

24.72 

17.57 

21.65 


AR(2)  Data  with  =  —0.5  and  (^2  =  —0.3 


None 


Step 

Ramp 

Exp 

Sine 


Step 

Ramp 

Exp 

Sine 


Step 

Ramp 

Exp 

Sine 


Step 

Ramp 

Exp 

Sine 


107.19 


212.61 

234.33 

236.64 

310.66 


290.06 

240.87 
253.94 

195.87 


194.39 

227.86 

255.36 

241.21 


219.75 

237.25 

282.18 

201.47 


46.84 


206.70 

198.47 

170.62 

451.15 


241.69 

181.80 

168.89 

266.29 


104.11 

133.96 

211.82 

352.77 


98.57 

102.55 

287.65 

211.65 


11.91 


19.16 

22.82 

14.73 

20.14 


14.81 

16.27 

11.19 

23.30 


9.78 

12.44 

11.55 

23.51 


7.55 

8.80 

13.66 

17.22 


2.49 


22.46 

47.54 

31.35 

24.97 


22.09 

46.06 

37.10 

26.72 


22.34 

48.91 

40.65 

26.85 


31.57 

44.67 

36.14 

24.93 


8.02 


23.41 

42.08 

30.84 

21.79 


22.40 

34.04 

25.27 

24.03 


15.44 

26.82 

24.62 

25.94 


10.20 

17.09 

29.03 

18.44 
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Table  35.  Fishman’s  Number  and  Size  of  Batches  for  AR(2)  with  Measurement  Noise  Model 


AR(2)  Data  with  <f>i 


Transient  Number  of  Batches 
Afiy{ay)  Type  fiM 


None 

2.0 

Step 

2.0 

Ramp 

2.0 

Exp 

2.0 

Sine 

1.5 

Step 

1.5 

Ramp 

1.5 

Exp 

1.5 

Sine 

1.0 

Step 

1.0 

Ramp 

1.0 

Exp 

1.0 

Sine 

0.5 

Step 

0.5 

Ramp 

0.5 

Exp 

0.5 

Sine 

AR(2) 

None 

2.0 

Step 

2.0 

Ramp 

2.0 

Exp 

2.0 

Sine 

1.5 

Step 

1.5 

Ramp 

1.5 

Exp 

1.5 

Sine 

1.0 

Step 

1.0 

Ramp 

1.0 

Exp 

1.0 

Sine 

0.5 

Step 

0.5 

Ramp 

0.5 

Exp 

0.5 

Sine 

60.34 


355.32 

228.93 

210.48 

151.46 


336.94 

257.46 

214.46 
179.62 


132.29 

151.41 

154.54 

259.38 


lata  wii 


25.03 


113.33 


65.98 

07.06 

147.53 

119.41 


33.48 

33.03 

123.14 

100.59 


115.29 

99.57 

107.20 

66.26 


239.43 

263.72 

227.98 

189.62 


255.74 

276.15 

218.30 

238.60 


43.17 

36.53 

148.30 

254.61 


=  -1.5 


9.58 


16.79 

87.70 

06.44 

103.97 


116.30 

92.99 

88.59 

120.04 


77.89 

100.87 

84.76 

108.34 


45.20 

48.23 

68.93 

54.48 


21.85 

13.33 

11.14 

16.50 

46.45 

55.52 

16.70 

31.95 

23.59 

19.57 

12.56 

19.84 

48.27 

59.28 

32.53 

49.95 

23.50 

19.67 

12.70 

21.06 

39.39 

56.80 

24.49 

43.57 

26.23 

24.31 

12.97 

19.14 

11.14 

15.95 

30.47 

46.47 

14.46 

13.83 

and  4>2  =  —0.8 

45.78 

17.21 

21.56 

25.43 

47.36 

54.97 

21.21 

24.76 

21.73 

16.05 

20.67 

24.63 

42.81 

53.61 

13.46 

17.65 

22.05 

15.93 

13.85 

15.78 

23.15 

34.57 

17.17 

22.94 

24.40 

14.74 

13.82 

15.69 

19.57 

24.21 

19.52 

22.99 

31.47 

19.36 
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filter  formulations,  the  truncation  point  selected  by  the  AR(2)  algorithm  generally  result 
in  postiv'  truncation-point  errors.  The  AR(2)  with  measurement  noise  forinulation  almost 
always  results  in  better  confidence  interval  coverage  rates  and  a  significantly  higher  pass  rate 
on  Schruben’s  initial  bias  test.  Since  these  tests  are  conducted  with  pure  AR(2)  data,  it  is 
not  surprising  that  the  AR(2)  with  measurement  noise  formulation  performs  better.  The 
improvement  may  also  be  attributed  to  the  increased  flexibility  of  the  AR(2)  with  measure¬ 
ment  noise  model.  Because  of  the  impro'^ed  performance  of  the  AR(2)  with  measurement 
noise  formulation,  only  that  model  is  applied  in  subsequent  tests. 


4-6.2  Autoregressive- Moving  Average  Test  Cases.  A  similar  Monte  Carlo  analysis  is 
conducted  on  data  sets  generated  from  first-order  autoregressive-moving  average  ARM A(  1,1) 
models.  The  ARMA(1,1)  model  is  selected  because  it  is  a  simple  time-series  model  from 
which  data  is  generated  easily,  but  the  ARM A(  1,1)  data  can  not  be  fit  precisely  with  an 
AR(1)  or  AR(2)  formulation  for  the  Kalman  filter.  Therefore,  these  data  sets  test  the 
robustness  of  the  approach.  Since  the  assumed  filter  model  does  not  fit  the  data,  some 
adjustment  for  “lack  of  fit”  is  necessary.  When  actual  discrete-event  simulation  output, 
which  has  an  unknown  analytical  formulation,  is  modeled,  the  truncation-point  selection 
algorithm  needs  to  perform  with  some  “lack  of  fit”.  The  ARMA(1,1)  model  used  to  generate 
the  data  is 

Vn-  =  MVn-l  -  A'y)  +  Wd{tn)  - 


with  variance  of 


4  = 


1  0 1  —  2(l>i0i 

.  w!  , 


Qa 


Table  36  shows  the  parameters  used  to  generate  the  data  sequences  {y„}.  These  parameters 
are  selected  to  test  a  wide  range  of  the  ARM A(  1,1)  stable  region. 


As  seen  in  Table  37,  for  some  combinations  of  parameters,  the  model  parameters  often 
failed  to  meet  the  model  estimation  criterion.  Equation  (58),  or  required  a  significant  number 
of  observations  to  estimate  the  model.  This  indicates  that  the  AR(2)  model  embedded  within 
the  Kalmpe  filter  is  unable  to  adjust  to  the  correlation  in  ARMA(1,1)  data. 
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Table  36.  ARMA(1,1)  Parameters  for  Generated  Data 


01 

-0.8 

+0.3 

-0.8 

-0.3 

-0.3 

+0.8 

-0.3 

-0.8 

+0.3 

+0.8 

+0.3 

-0.8 

+0.8 

+0.3 

+0.8 

-0.3 

Table  37.  Kalman  Filter  Model  Estimation  Summary 


Data  Parameters 

Fail 

Observations  Used  To  Estimate 
Min  Max  Avg  St  Dev 

Cusum  Alarms 

Min  Max  Avg  St  Dev 

ARM A(  1,1)  Data  with  AR(2)  with  Measurement  Noise  Model 

-0.8 

01  =  0.3 

0.324 

296 

1499 

1154.7 

226.1 

0 

10 

2.2 

0.7 

IB 

-0.8 

6x  =  -0.3 

267 

1499 

904.9 

390.6 

0 

20 

3.8 

2.3 

=  0.8 

470 

1491 

1230.1 

124.2 

2.1 

-0.3 

Ox  =  -0.8 

256 

1499 

516.0 

303.6 

21 

6.2 

2.9 

<t>i  ~ 

0.3 

9x  =  0.8 

346 

1499 

1188.5 

145.5 

■a 

2.1 

0.8 

4>i  = 

0.3 

6x  =  -0.8 

Eia 

253 

1414 

335.0 

105.6 

la 

3.1 

^1  = 

0.8 

Ox  =  0.3 

0.001 

252 

1492 

306.8 

74.8 

(f>\  = 

0.8 

dx  =  -0.3 

0.044 

252 

1494 

385.3 

Note:  Each  row  represents  16,000  runs;  1,000  for  each  transient  type  and  magnitude. 


Table  38  shows  the  estimated  parameters  for  fitting  an  AR(2)  model  with  measurement 
noise  to  ARMA(1,1)  data.  Based  on  the  parameter  estimates,  the  variance  estimates  in 
Table  39  are  calculated.  The  variance  of  the  measurement  noise  R  should  represent  the 
“lack  of  fit”.  Since  the  ARM A(  1,1)  data  is  generated  with  dynamics  noise  variance  of 
1.0,  the  dynamics  noise  is  slightly  underestimated. 

Tables  40  through  43  show  the  selected  truncation-point  errors,  [ho  —  no).  The  average 
truncation  error  is  negative,  meaning  that  generally  the  estimated  truncation  point  no  retains 
transient  data  in  the  supposedly  steady-state  sequence.  However,  for  the  ARM A(  1,1)  case 
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Table  38.  AR(2)  with  Measurement  Noise  Model  Parameter  Estimation  Summary 


Data  Parameters 

■ 

■Bl 

Hi 

mm 

Hi 

■ 

■ 

ARM A(  1,1)  Data  with  AR(2)  with  Measurement  Noise  Model 

4>i 

=  -0.8 

CO 

o 

II 

9.93 

0.09 

-1.19 

0.14 

-0.33 

0.12 

0.77 

0.14 

4>\ 

=  -0.8 

II 

1 

p 

9.96 

0.07 

-0.56 

0.19 

0.19 

0.16 

0.78 

0.12 

=  -0.3 

=  0.8 

9.91 

0.09 

-0.90 

0.10 

-0.33 

0.07 

0.74 

0.19 

<l>i 

=  -0.3 

00 

o 

1 

II 

rH 

9.99 

0.08 

0.44 

0.07 

-0.24 

0.06 

0.90 

0.00 

=  0.3 

00 

o 

II 

9.92 

0.09 

-0.38 

0.19 

-0.16 

0.17 

0.68 

0.29 

<t>\ 

=  0.3 

II 

1 

O 

bo 

10.00 

0.14 

0.91 

0.06 

-0.39 

0.05 

0.90 

0.00 

<l>\ 

=  0.8 

CO 

o 

II 

10.00 

0.20 

0.67 

0.13 

0.08 

0.10 

0.73 

0.12 

4>i 

=  0.8 

=  -0.3 

10.00 

0.35 

1.17 

0.06 

-0.35 

0.06 

0.90 

0.00 

Note:  Each  row  represents  the  runs  that  did  not  fail  to  estimate  a  model  from  16,000  runs. 


Table  39.  AR(2)  with  Measurement  Noise  Model  Variance  Estimation  Summary 

I - ^ - 1 - " - - - 1 - * - n - » 


Data  Parameters 


Qd 


R 


f^R 


ARMA(1,1)  Data  with  AR(2)  with  Measuremen 


'ay 


Noise  Model 


-0.8 

^1  = 

0.73 

0.23 

0.39 

0.36 

2.12 

0.21 

2.09 

-0.8 

^1  = 

bh 

0.76 

0.23 

0.22 

0.12 

1.32 

0.08 

1.30 

<t>l  = 

-0.3 

^1  = 

0.8 

0.98 

0.26 

0.52 

0.53 

1.58 

0.10 

1.53 

<t>l  = 

-0.3 

-0.8 

0.96 

0.08 

0.11 

0.02 

1.13 

0.05 

1.13 

<t>\  = 

0.3 

0.8 

0.89 

0.50 

0.52 

0.68 

1.23 

0.16 

1.13 

4>\  = 

0.3 

-0.8 

0.99 

0.09 

0.12 

0.01 

1.48 

0.09 

1.53 

4>\  = 

0.8 

^1  = 

0.3 

0.63 

0.17 

0.27 

0.12 

1.28 

0.09 

1.30 

<t>i  = 

0.8 

e,= 

-0.3 

0.78 

0.07 

0.10 

0.01 

2.00 

0.72 

2.09 

:h  row 

represents  the 

runs  that  did  not  : 

ail  to  estima 

,e  a  model  from  16 

000  runs. 
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Table  40.  AR(2)  Truncation  Point  Errors  (ho  —  tIq 


^Mv(o'v)  Type 


2.0 

Step 

2.0 

Ramp 

2.0 

Exp 

2.0 

Sine 

1.5 

Step 

1.5 

Ramp 

1.5 

Exp 

1.5 

Sine 

1.0 

Step 

1.0 

Ramp 

1.0 

Exp 

1.0 

Sine 

0.5 

Step 

0.5 

Ramp 

0.5 

Exp 

1  0.5 

Sine 

0.104 


2.0 

Step 

2.0 

Ramp 

2.0 

Exp 

2.0 

Sine 

1.5 

Step 

1.5 

Ramp 

1.5 

Exp 

1.5 

Sine 

1.0 

Step 

1.0 

Ramp 

1.0 

Exp 

1.0 

Sine 

0.5 

Step 

0.5 

Ramp 

0.5 

Exp 

0.5 

Sine 

0.000 

0.034 


0.039 


.443 
.042 
0.000 
0.062 


Min 

Max 

Avg 

St  Dev 

MSE 

lata  with  4>i  = 

=  —0.8  and  6^ 

=  0.3 

-150 

56 

-57.5 

44.5 

5284.4 

-290 

41 

-154.7 

75.8 

29669.4 

-248 

53 

-103.5 

70.1 

15613.1 

-480 

40 

-193.3 

142.1 

57570.4 

-211 

41 

-83.7 

60.7 

10698.0 

-340 

26 

-174.0 

86.2 

37701.6 

-314 

44 

-139.9 

90.4 

27748.6 

-485 

17 

-256.3 

155.3 

89815.6 

-366 

30 

-133.4 

97.5 

27311.1 

-432 

28 

-218.6 

103.1 

58397.9 

-392 

10 

-187.6 

96.8 

44547.1 

-486 

-4 

-315.5 

137.2 

118340.8 

-471 

33 

-170.9 

132.1 

46639.9 

-486 

-33 

-282.3 

100.5 

89819.9 

-464 

-22 

-293.2 

88.9 

93881.2 

-486 

-124 

-366.3 

102.3 

144627.6 

ata  with  <f)i  = 

—0.8  and  6i 

=  -0.3 

-447 

80 

-57.8 

82.1 

10074.9 

-401 

398 

-137.3 

96.0 

28063.3 

-371 

410 

-105.4 

96.2 

20377.5 

-483 

178 

-212.2 

153.6 

68605.9 

-466 

62 

-68.6 

100.4 

14776.9 

-454 

-7 

-165.2 

98.2 

36931.9 

-409 

216 

-162.1 

92.0 

34752.8 

-483 

25 

-250.0 

127.7 

78812.6 

-467 

56 

-79.2 

110.5 

18482.4 

-476 

494 

-205.8 

99.5 

52251.7 

-450 

639 

-218.1 

86.0 

54982.9 

-483 

447 

-296.6 

105.2 

99032.2 

-461 

288 

-177.4 

135.8 

49910.7 

-477 

372 

-309.3 

85.9 

103019.6 

-491 

460 

-297.9 

78.4 

94900.2 

-484 

489 

-336.5 

78.6 

119409.4 
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Table  41.  AR(2)  Truncation  Point  Errors  {hp  —  tIq) 


Fail 

Min 

Max 

Avg 

St  Dev 

MSE 

ARMA(1,1)  Data  with 

=  —0.3  and  di 

=  0.8 

2.0 

Step 

0.000 

-139 

-17 

-69.1 

39.2 

6307.4 

2.0 

Ramp 

0.000 

-280 

-86 

-171.9 

58.6 

32987.2 

2.0 

Exp 

0.000 

-259 

-48 

-136.6 

65.7 

22989.9 

2.0 

Sine 

-485 

-34 

-245.9 

118.4 

74463.5 

1.5 

Step 

0.000 

-194 

-24 

-104.4 

55.0 

13929.6 

1.5 

Ramp 

0.000 

-328 

-100 

-202.1 

69.2 

45622.7 

1.5 

Exp 

0.000 

-317 

-58 

-171.9 

72.9 

34875.7 

1.5 

Sine 

-485 

-64 

-342.8 

80.4 

123989.7 

1.0 

Step 

-324 

-28 

-153.1 

92.2 

31950.2 

1.0 

Ramp 

-421 

-76 

-242.4 

87.0 

66317.2 

1.0 

Exp 

-386 

-123 

-233.5 

74.0 

60019.6 

1.0 

Sine 

IHiiin 

-486 

-187 

-409.9 

60.5 

171717.8 

0.5 

Step 

0.338 

-465 

-42 

-142.4 

81.4 

26891.8 

0.5 

Ramp 

0.001 

-478 

-189 

-304.6 

86.4 

100269.6 

0.5 

Exp 

0.000 

-444 

-227 

-322.0 

59.6 

107244.4 

0.5 

Sine 

0.009 

-487 

-313 

-441.5 

26.4 

195630.4 

ARM A(  1,1)  Data  with  <f>i  = 

—0.3  and  0i 

=  -0.8 

2.0 

Step 

inmii 

-403 

593 

3.5 

71.9 

5184.6 

2.0 

Rcimp 

-372 

631 

-94.8 

87.2 

16596.8 

2.0 

Exp 

-321 

677 

-89.7 

88.5 

15874.2 

2.0 

Sine 

HI 

-466 

611 

-155.0 

106.3 

35307.0 

1.5 

Step 

-431 

704 

-1.9 

89.5 

8022.5 

1.5 

Ramp 

Hil 

-394 

687 

-127.0 

98.6 

25847.3 

1.5 

Exp 

0.000 

-380 

689 

-143.5 

95.7 

29761.7 

1.5 

Sine 

0.001 

-474 

725 

-204.3 

118.9 

55883.6 

Step 

-441 

647 

-42.6 

103.7 

12568.0 

Ramp 

ilflllfl 

-457 

-182.4 

113.4 

46143.7 

Exp 

0.000 

-427 

715 

-201.1 

110.4 

52645.8 

Sine 

0.011 

-477 

654 

-256.2 

116.9 

79285.7 

0.5 

Step 

-460 

637 

-148.7 

181.0 

54893.1 

0.5 

Ramp 

Eia 

-479 

698 

-295.1 

151.4 

110006.7 

0.5 

Exp 

0.001 

-488 

647 

-265.4 

112.4 

83057.0 

0.5 

Sine 

0.009 

-482 

624 

-310.2 

116.0 

109704.9 

Note:  Negative  Truncation  points  indicate  transient  observations  are  retained. 
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Table  42.  AR(2)  Truncation  Point  Errors  {uo  —  n, 


Type 


2.0 

Step 

2.0 

Ramp 

2.0 

Exp 

2.0 

Sine 

1.5 

Step 

1.5 

Ramp 

1.5 

Exp 

1.5 

Sine 

1.0 

Step 

1.0 

Ramp 

1.0 

Exp 

1.0 

Sine 

0.5 

Step 

0.5 

Ramp 

0.5 

Exp 

0.5 

Sine 

0.000 

0.000 

0.000 

0.010 


Min 

Max 

Avg 

St  Dev 

MSE 

)ata  with  (f>i  = 

=  0.3  and  6i 

=  0.8 

-102 

20 

-53.3 

28.8 

3673.7 

-238 

17 

-136.2 

54.0 

21462.2 

-233 

8 

-112.6 

58.3 

16081.4 

-479 

1 

-205.1 

117.6 

55901.8 

-145 

11 

-71.7 

40.3 

6757.8 

-281 

-7 

-157.6 

61.8 

28647.6 

-287 

5 

-149.7 

67.2 

26934.9 

-488 

0 

-276.1 

104.4 

87138.5 

-227 

5 

-103.4 

63.8 

14776.6 

-360 

-15 

-195.2 

73.8 

43543.0 

-345 

-36 

-198.2 

65.8 

43622.4 

-488 

-78 

-333.7 

82.2 

118141.3 

-468 

8 

-149.6 

103.8 

33160.2 

-482 

-108 

-262.7 

86.4 

76462.5 

-408 

-73 

-287.2 

59.5 

86012.8 

-489 

-183 

-382.4 

70.7 

151217.3 

0.109 


ARMA(1,1)  Data  with  =  0.3  and  6i  =  —0.8 


2.0 

Step 

0.000 

2.0 

Ramp 

0.000 

2.0 

Exp 

0.000 

2.0 

Sine 

0.000 

1.5 

Step 

0.000 

1.5 

Ramp 

0.000 

1.5 

Exp 

0.000 

1.5 

Sine 

0.000 

1.0 

Step 

0.000 

1.0 

Ramp 

0.000 

1.0 

Exp 

0.000 

1.0 

Sine 

0.000 

0.5 

Step 

0.5 

Ramp 

0.5 

Exp 

■Siiiira 

0.5 

Sine 

0.000 

41.1 

-26.6 

22.8 

-15.0 


124.8 

133.8 
133.2 
146.5 


17254.1 

18601.0 

18272.7 

21682.0 


106.6 

12369.2 

153.8 

25294.4 

142.0 

21327.7 

157.2 

31579.0 

124.2 

16360.0 

164.0 

32889.7 

164.5 

37723.1 

190.0 

57842.3 

183.5 

35583.3 

200.5 

75740.0 

195.7 

70980.5 

200.8 

91777.4 
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Table  43.  AR(2)  Truncation  Point  Errors  (no  —  rio 


2.0 

Step 

0.000 

2.0 

Ramp 

0.000 

2.0 

Exp 

0.000 

2.0 

Sine 

0.000 

1.5 

Step 

0.000 

1.5 

Rtimp 

0.000 

1.5 

Exp 

0.000 

1.5 

Sine 

0.000 

1.0 

Step 

0.000 

1.0 

Ramp 

0.000 

1.0 

Exp 

0.000 

1.0 

Sine 

0.001 

0.5 

Step 

0.000 

0.5 

Ramp 

0.000 

0.5 

Exp 

0.000 

1  0.5 

Sine 

0.000 

Avg 

St  Dev 

MSE 

=  0.8  and  6i 

=  0.3 

23.4 

89.3 

8528.1 

-13.7 

90.9 

8453.2 

19.9 

86.6 

7895.0 

9.7 

96.3 

9374.1 

19.6 

76.5 

6236.7 

-24.7 

92.7 

9204.8 

0.0 

102.4 

10480.5 

-33.5 

121.9 

15978.2 

21.9 

92.3 

8989.9 

-42.9 

118.7 

15937.5 

-67.5 

119.3 

18799.5 

-110.5 

132.5 

29777.7 

84.2 

7227.8 

125.2 

30183.2 

144.5 

44410.2 

1  158.1 

64036.8 

ARM A(  1,1)  Data  with  =  0.8  and  B-i  =  —0.3 


0.000 

0.000 

0.000 

0.001 


1 

0 

0 

3 


6 
0 

0.000 

0.003 


5 
0 

0.000 

0.002 


Note:  Negative  Truncation  points  indicate  transient  observations  are  retained 


2.0 

Step 

2.0 

Reimp 

2.0 

Exp 

2.0 

Sine 

1.5 

Step 

1.5 

Ramp 

1.5 

Exp 

1.5 

Sine 

1.0 

Step 

1.0 

Ramp 

1.0 

Exp 

1.0 

Sine 

0.5 

Step 

0.5 

Ramp 

0.5 

Exp 

0.5 

Sine 

56.6 

146.6 

24688.3 

27.9 

171.3 

30138.0 

57.0 

155.9 

27547.7 

40.5 

153.9 

25323.9 

52.9 

145.8 

24055.8 

13.8 

168.3 

28500.9 

47.9 

153.7 

25925.7 

31.2 

180.0 

33367.4 

54.4 

152.4 

26185.5 

-2.8 

182.6 

33343.7 

-0.6 

169.3 

28657.3 

-42.0 

205.6 

44038.4 

52.7 

168.4 

31147.2 

-72.9 

200.8 

45636.2 

-90.9 

216.5 

55140.1 

-136.4 

241.4 

76865.1 
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with  <f>i  =  0.8  amd  6i  =  —0.3,  the  algorithm  selected  good  truncation  points  ho  since  the 
errors  are  on  the  average  are  about  50  observations  after  the  completion  of  the  induced 
transient. 

For  the  ARMA(1,1)  data,  the  coverage  rate  is  less  for  the  sequence  truncated  at  the 
estimated  truncation  point  no  compared  to  the  sequence  truncation  at  the  true  truncation 
point  no.  As  before,  coverage  decreaises  as  the  shift  in  the  mean  decreases  with  the  exception 
of  the  exponentially  decaying  sinusoidad  transient.  For  that  transient,  low  transient  values 
and  high  transient  values  may  cancel  sufEciently  to  prevent  a  significant  reduction  in  coverage 
rate.  The  decaying  exponential  sinusoidal  transient  also  has  a  very  high  rate  of  passing 
Schruben’s  test  for  initial  bias.  The  mixture  of  low  and  high  transient  values  are  apparently 
more  difficult  for  that  statistical  test  to  detect. 

The  extent  of  the  coverage  decrease  for  the  sequence  truncated  at  the  estimated  point 
appears  to  depend  on  the  correlation  structure  in  the  generated  data.  For  the  data  sets 
with  a  positive  theoretical  autocorrelation  at  lag  one  (see  Table  36),  the  decrease  in  coverage 
is  small.  For  the  same  cases,  the  truncated  sequences  pass  Schruben’s  test  for  initial  bias 
at  fairly  high  rates.  Actucil  discrete-event  simulation  output  sequences  generally  have  high 
positive  correlation,  so  these  results  are  promising. 

Tables  48  through  51  present  the  number  and  size  of  the  batches  used  in  the  confidence 
interval  construction  techniques.  For  each  case,  these  numbers  for  1000  sequences  truncated 
at  the  true  truncation  point  no  are  also  shown.  In  general,  the  sequences  truncated  at  the 
estimated  point  has  about  the  same  number  of  batches  and  batch  sizes.  For  the  four  cases 
with  positive  autocorrelation  at  lag  one,  the  case  with  =  —0.3  and  0i  —  0.8  has  the  largest 
decrease  in  coverage.  For  this  case,  the  number  of  batches  increases  dramatically  from  the 
sequences  without  a  transient.  The  increase  in  the  number  of  batches  causes  tight  confidence 
intervals  about  a  slightly  biased  mean  estimate. 

4-6.3  Analysis.  The  Monte  Carlo  results  indicate  that  the  technique  works  well  for 
sequences  which  display  a  high  positive  autocorrelation  and  significant  transients.  The  al- 
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Table  44.  AR(2)  with  Mecisurement  Noise  Model  Truncation  Point  hp  Evaluations 


Transient 
Afiy{ay)  Type 

Fail 

Cov 

NOBM 

Half  Width 

Cov 

OBM 

Half  Width 

Bieis  Test 

A 

EIMA(1,1)  Data  with  <f>i  =  — 

0.8  and  9i  =  0.3 

None 

0.02 

0.94 

0.027  ±  0.009 

0.95 

0.028  ±  0.008 

1.00 

2.0 

Step 

0.65 

0.052  ±  0.051 

0.75 

0.062  ±  0.082 

0.24 

2.0 

Ramp 

0.41 

0.089  ±  0.107 

0.52 

0.101  ±  0.141 

0.49 

2.0 

Exp 

0.51 

0.39 

0.041  ±  0.012 

0.57 

0.049  ±  0.032 

0.31 

2.0 

Sine 

0.01 

0.82 

0.091  ±  0.105 

0.90 

0.098  ±  0.099 

0.89 

1.5 

Step 

0.78 

0.050  ±  0.035 

0.70 

0.048  ±  0.039 

0.23 

1.5 

Ramp 

IM 

0.33 

0.059  ±  0.066 

0.46 

0.067  ±  0.091 

0.57 

1.5 

Exp 

0.43 

0.38 

0.041  ±  0.038 

0.38 

0.041  ±  0.038 

0.49 

1.5 

Sine 

0.00 

0.90 

0.090  ±  0.096 

0.96 

0.096  ±  0.091 

0.86 

1.0 

Step 

w 

0.44 

0.042  ±  0.016 

0.63 

0.050  ±  0.034 

MEM 

1.0 

R2mip 

il9 

0.30 

0.040  ±  0.026 

0.36 

0.043  ±  0.042 

■IB 

1.0 

Exp 

la 

0.31 

0.037  ±  0.016 

0.33 

0.038  ±  0.023 

0.71 

1.0 

Sine 

0.00 

0.76 

0.065  ±  0.076 

0.98 

0.085  ±  0.071 

0.86 

0.5 

Step 

Esa 

0.28 

0.037  ±  0.006 

0.28 

0.038  ±  0.006 

MEM 

0.5 

Ramp 

0.22 

0.035  ±  0.005 

0.23 

0.035  ±  0.006 

0.5 

Exp 

la 

0.27 

0.034  ±  0.007 

0.27 

0.034  ±  0.010 

■B 

0.5 

Sine 

0.00 

0.80 

0.053  ±  0.059 

0.97 

0.067  ±  0.055 

AF 

1,MA(1,1)  Data  with  <pi  =  — 0 

.8  and  0i  =  —0.3 

None 

0.00 

0.92 

0.041  ±  0.003 

0.93 

0.041  ±  0.003 

0.93 

2.0 

Step 

EEa 

0.83 

0.048  ±  0.020 

0.85 

0.049  ±  0.027 

0.42 

2.0 

Ramp 

la 

0.70 

0.054  ±  0.046 

0.73 

0.057  ±  0.062 

0.24 

2.0 

Exp 

0.20 

0.68 

0.050  ±  0.045 

0.69 

0.051  ±  0.052 

0.26 

2.0 

Sine 

0.00 

0.86 

0.066  ±  0.073 

0.91 

0.075  ±  0.077 

0.58 

1.5 

Step 

mi 

0.82 

0.047  ±  0.035 

0.86 

0.050  ±  0.035 

1.5 

Ramp 

0.67 

0.045  ±  0.026 

0.69 

0.046  ±  0.032 

1.5 

Exp 

Mg 

0.54 

0.043  ±  0.020 

0.54 

0.043  ±  0.022 

0.10 

1.5 

Sine 

0.00 

0.85 

0.055  ±  0.055 

0.91 

0.065  ±  0.061 

0.62 

1.0 

Step 

0.77 

0.044  ±  0.027 

0.78 

0.045  ±  0.024 

MEM 

1.0 

Reimp 

ta 

0.57 

0.041  ±  0.013 

0.58 

0.042  ±  0.018 

■SB 

1.0 

Exp 

0.15 

0.51 

0.039  ±  0.011 

0.52 

0.040  ±  0.016 

0.11 

1.0 

Sine 

0.00 

0.87 

0.051  ±  0.049 

0.93 

0.057  ±  0.051 

0.52 

0.5 

Step 

0.49 

0.040  ±  0.008 

0.49 

0.040  ±  0.008 

0.5 

Ramp 

■nBfi 

0.37 

0.037  ±  0.006 

0.37 

0.037  ±  0.006 

0.5 

Exp 

0.15 

0.58 

0.038  ±  0.011 

0.58 

0.039  ±  0.016 

0.5 

Sine 

0.00 

0.87 

0.045  ±  0.035 

0.91 

0.047  ±  0.033 

0.75 

Note:  Since  the  nominal  rates  are  0.9,  the  coverage  and  bias  test  accuracies  are  ^  ±0.016 


for  1000  runs. 
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Table  45.  AR(2)  with  Measurement  Noise  Model  Truncation  Point  hp  Evaluations 


Transient 
Any{ay)  Type 

Fail 

Cov 

NORM 

Half  Width 

Cov 

OEM 

Half  Width 

Bias  Test 

nn) 

A 

[IMA(1,1)  Data  with  <f>i  —  — 

0.3  and  6i  —  0.8 

None 

0.12 

0.98 

0.013  ±  0.004 

0.98 

0.014  ±  0.003 

0.99 

2.0 

Step 

mi 

0.11 

0.050  ±0.030 

0.16 

0.058  ±  0.046 

2.0 

Ramp 

Era 

0.02 

0.077  ±  0.091 

0.22 

0.098  ±  0.154 

2.0 

Exp 

Ell 

0.00 

0.044  ±  0.010 

0.07 

0.046  ±  0.011 

2.0 

Sine 

0.02 

0.51 

0.083  ±  0.123 

0.93 

0.109  ±  0.097 

1.5 

Step 

0.24 

0.057  ±  0.016 

0.09 

0.054  ±  0.011 

1.5 

Ramp 

Era 

0.04 

0.062  ±  0.059 

0.20 

0.067  ±  0.075 

BSH 

1.5 

Exp 

0.75 

0.01 

0.040  ±  0.021 

0.01 

0.041  ±  0.025 

BEH 

1.5 

Sine 

0.00 

0.72 

0.065  ±  0.090 

0.96 

0.092  ±  0.092 

1.0 

Step 

0.02 

0.048  ±  0.008 

0.06 

0.049  ±  0.012 

1.0 

Ramp 

0.55 

0.065  ±  0.041 

0.40 

0.053  ±  0.028 

1.0 

Exp 

Era 

0.19 

0.043  ±  0.029 

0.15 

0.040  ±  0.024 

1.0 

Sine 

0.00 

0.58 

0.087  ±  0.116 

0.99 

0.125  ±  0.079 

0.5 

Step 

mi 

0.02 

0.036  ±  0.005 

0.08 

0.037  ±  0.007 

0.5 

Ramp 

Era 

0.01 

0.035  ±  0.012 

0.25 

0.043  ±  0.029 

BEH 

0.5 

Exp 

0.53 

0.00 

0.035  ±  0.015 

0.02 

0.036  ±  0.015 

BSH 

0.5 

Sine 

0.00 

0.64 

0.073  ±  0.080 

0.99 

0.103  ±  0.049 

Af 

LMA(1,1)  Data  with  <f>i  =  — 0 

.3  and  =  —0.8 

None 

0.00 

0.88 

0.070  ±  0.007 

0.88 

0.069  ±  0.006 

0.85 

2.0 

Step 

0.85 

0.072  ±  0.020 

0.85 

0.072  ±  0.023 

WWM 

2.0 

Ramp 

lifwl 

0.76 

0.071  ±  0.025 

0.76 

0.071  ±  0.027 

BH 

2.0 

Exp 

0.04 

0.73 

0.075  ±  0.037 

0.72 

0.074  ±  0.033 

2.0 

Sine 

0.00 

0.85 

0.074  ±  0.035 

0.86 

0.076  ±  0.038 

0.66 

1.5 

Step 

0.87 

0.072  ±  0.027 

0.87 

0.072  ±  0.025 

mm 

1.5 

Ramp 

0.72 

0.072  ±  0.026 

0.72 

0.072  ±  0.027 

1.5 

Exp 

0.03 

0.64 

0.072  ±  0.028 

0.63 

0.071  ±  0.025 

■gra 

1.5 

Sine 

0.00 

0.86 

0.071  ±  0.028 

0.86 

0.073  ±  0.035 

0.62 

1.0 

Step 

0.82 

0.075  ±  0.037 

0.81 

0.074  ±  0.033 

0.54 

1.0 

Ramp 

lifirl 

0.65 

0.070  ±  0.023 

0.64 

0.070  ±  0.024 

0.21 

1.0 

Exp 

0.03 

0.67 

0.068  ±  0.020 

0.67 

0.068  ±  0.020 

0.22 

1.0 

Sine 

0.00 

0.85 

0.068  ±  0.025 

0.86 

0.070  ±  0.029 

0.68 

0.5 

Step 

0.03 

0.64 

0.070  ±  0.028 

0.63 

0.070  ±  0.025 

0.29 

0.5 

Ramp 

0.05 

0.56 

0.065  ±  0.019 

0.56 

0.065  ±  0.018 

0.14 

0.5 

Exp 

0.03 

0.73 

0.065  ±  0.015 

0.73 

0.064  ±  0.015 

0.28 

0.5 

Sine 

0.00 

0.85 

0.065  ±  0.020 

0.85 

0.066  ±  0.020 

0.73 

Note;  Since  the  nominal  rates  are  0.9,  the  coverage  and  bias  test  accuracies  are  r;  ±0.016 


for  1000  runs. 
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Table  46.  AR(2)  with  Measurement  Noise  Model  Truncation  Point  ho  Evaluations 


Transient 
Any{ay)  Type 

Fail 

Cov 

NORM 

Half  Width 

Cov 

OEM 

Half  Width 

Bias  Test 

m) 

IRMA 

[1,1)  Data  with  =  C 

.3  and  0i  =  0.8 

None 

0.03 

0.97 

0.020  ±  0.005 

0.98 

0.020  ±  0.004 

1.00 

2.0 

Step 

0.51 

0.22 

0.057  ±  0.026 

0.26 

0.059  ±  0.030 

2.0 

Ramp 

0.17 

0.22 

0.072  ±  0.109 

0.23 

0.067  ±  0.088 

2.0 

Exp 

0.18 

0.11 

0.046  ±  0.014 

0.13 

0.047  ±  0.015 

BsH 

2.0 

Sine 

0.01 

0.57 

0.064  ±  0.116 

0.84 

0.100  ±  0.143 

— 

1.5 

Step 

W 

0.13 

0.057  ±  0.023 

0.17 

0.059  ±  0.027 

1.5 

Ramp 

Eig 

0.12 

0.056  ±  0.061 

0.16 

0.059  ±  0.078 

1.5 

Exp 

0.37 

0.12 

0.045  ±  0.034 

0.14 

0.047  ±  0.037 

HH 

1.5 

Sine 

0.01 

0.61 

0.066  ±  0.119 

0.85 

0.098  ±  0.128 

1.0 

Step 

0.46 

0.19 

0.055  ±  0.017 

0.20 

0.055  ±  0.018 

■EH 

1.0 

Ramp 

0.29 

0.14 

0.050  ±  0.043 

0.19 

0.053  ±  0.055 

1.0 

Exp 

0.26 

0.22 

0.047  ±  0.033 

0.18 

0.046  ±  0.037 

BH 

1.0 

Sine 

0.00 

0.69 

0.072  ±  0.127 

0.84 

0.090  ±  0.125 

■■ 

0.5 

Step 

Ha 

0.13 

0.040  ±  0.008 

0.18 

0.042  ±  0.010 

■EH 

0.5 

Ramp 

Ha 

0.19 

0.045  ±  0.023 

0.30 

0.048  ±  0.031 

0.5 

Exp 

0.20 

0.06 

0.036  ±  0.016 

0.13 

0.042  ±  0.021 

0.5 

Sine 

0.00 

0.83 

0.079  ±  0.129 

0.91 

0.093  ±  0.113 

■■ 

A 

EIMA(1,1)  Data  with  =  0.3  and  6i  =  —0.8 

None 

0.00 

0.88 

0.127  ±  0.020 

0.88 

0.127  ±  0.018 

0.79 

2.0 

Step 

0.00 

0.86 

0.130  ±  0.027 

0.86 

0.130  ±  0.025 

0.84 

2.0 

Ramp 

0.01 

0.83 

0.130  ±  0.033 

0.82 

0.129  ±  0.030 

0.56 

2.0 

Exp 

0.01 

0.85 

0.132  ±  0.033 

0.85 

0.131  ±  0.032 

0.64 

2.0 

Sine 

0.01 

0.83 

0.133  ±  0.031 

0.83 

0.133  ±  0.031 

0.76 

1.5 

Step 

0.00 

0.87 

0.129  ±  0.026 

0.86 

0.128  ±  0.024 

0.84 

1.5 

Ramp 

0.80 

0.132  ±  0.042 

0.79 

0.130  ±  0.034 

0.49 

1.5 

Exp 

IoprI 

0.77 

0.136  ±  0.051 

0.77 

0.134  ±  0.042 

0.40 

1.5 

Sine 

0.00 

0.83 

0.132  ±  0.033 

0.84 

0.131  ±  0.032 

0.70 

Step 

0.01 

0.86 

0.130  ±  0.028 

0.86 

0.129  ±  0.027 

Ramp 

0.01 

0.77 

0.133  ±  0.047 

0.77 

0.132  ±  0.043 

■B 

Exp 

0.01 

0.73 

0.133  ±  0.045 

0.73 

0.132  ±  0.043 

■Q 

Sine 

0.00 

0.85 

0.129  ±  0.031 

0.85 

0.128  ±  0.031 

0.62 

0.5 

Step 

0.76 

0.138  ±  0.055 

0.76 

0.136  ±  0.050 

0.5 

Ramp 

Itfis 

0.68 

0.133  ±  0.053 

0.68 

0.131  ±  0.050 

■H 

0.5 

Exp 

0.02 

0.75 

0.125  ±  0.040 

0.75 

0.124  ±  0.035 

0.29 

0.5 

Sine 

0.00 

0.84 

0.122  ±  0.031 

0.84 

0.121  ±  0.031 

0.76 

Note:  Since  the  nominal  rates  are  0.9,  the  coverage  and  bias  test  accuracies  are  ~  ±0.016 

for  1000  runs. 
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Table  47.  AR(2)  with  Measurement  Noise  Model  Truncation  Point  ho 


Transient 
A/iy(a„)  Type 


None 

2.0 

Step 

2.0 

Ramp 

2.0 

Exp 

2.0 

Sine 

1.5 

Step 

1.5 

Reimp 

1.5 

Exp 

1.5 

Sine 

1.0 

Step 

1.0 

Ramp 

1.0 

Exp 

1.0 

Sine 

0.5 

Step 

0.5 

Ramp 

0.5 

Exp 

0.5 

Sine 

A] 

None 

2.0 

Step 

2.0 

Ramp 

2.0 

Exp 

2.0 

Sine 

1.5 

Step 

1.5 

Ramp 

1.5 

Exp 

1.5 

Sine 

1.0 

Step 

1.0 

Ramp 

1.0 

Exp 

1.0 

Sine 

0.5 

Step 

0.5 

Ramp 

0.5 

Exp 

0.5 

Sine 

NOBM  OBM 

Cov  Half  Width  Cov  Half  Width 


ARMA(1,1)  Data  with  <f)i  -  0.8  and  6i  =  0.3 


0.01 


Evaluations 


Bias  Test 

nH) 


0.86  0.169  ifc  0.046 

0.85  0.164  ±  0.039 

0.85  0.170  ±  0.047 
0.84  0.168  ±  0.047 
0.85  0.169  ±  0.048 
0.83  0.172  ±  0.054 

0.85  0.166  ±  0.040 
0.83  0.165  ±  0.042 
0.85  0.165  ±  0.042 
0.83  0.170  ±  0.053 

0.86  0.168  ±  0.045 
0.82  0.170  ±  0.051 
0.81  0.175  ±  0.056 
0.82  0.182  ±  0.055 

0.84  0.164  ±  0.040 
0.81  0.168  ±  0.045 
0.80  0.172  ±  0.053 
0.83  0.183  ±  0.056 

0.84  0.169  ±  0.049 
0.81  0.176  ±  0.064 
0.78  0.193  ±  0.098 
0.86  0.181  ±  0.052 

0.83  0.165  ±  0.042 
0.80  0.172  ±  0.053 
0.77  0.188  ±  0.078 
0.86  0.181  ±  0.051 

0.84  0.172  ±  0.070 
0.74  0.199  ±  0.101 
0.75  0.186  ±  0.080 
0.85  0.170  ±  0.049 

0.83  0.169  ±  0.062 
0.75  0.194  ±  0.095 
0.74  0.183  ±  0.073 
0.85  0.169  ±  0.046 

ARMA(1,1)  Data  with  =  0.8  and  9i  =  —0.3 


mi 


0.87  0.315  ±  0.087 


0.82  0.319  ±  0.101 
0.82  0.315  ±  0.115 
0.85  0.315  ±  0.097 
0.84  0.334  ±  0.229 


0.84  0.314  ±  0.094 
0.80  0.324  ±  0.147 
0.82  0.322  ±  0.156 
0.82  0.338  ±  0.198 


0.84  0.318  ±  0.105 
0.81  0.324  ib  0.125 

0.80  0.339  ±  0.156 
0.83  0.342  ±  0.172 


0.82  0.325  ±  0.143 
0.7^^  0.343  ±  0.158 
0.75  0.341  ±  0.148 
0.83  0.315  ±  0.108 


Note:  Since  the  nominal  rates  are  0.9,  the  coverage  and  bias  test  accuracies  are 

for  1000  runs. 


0.306  ±  0.075 


0.311  ±  0.097 
0.311  ±  0.113 
0.307  ±  0.087 
0.329  ±  0.236 


0.308  ±  0.096 
0.318  ±  0.128 
0.313  ±  0.135 
0.336  ±  0.213 


0.310  ±  0.096 
0.319  ±  0.119 
0.333  ±  0.135 
0.344  ±  0.197 


0.317  ±  0.128 
0.337  ±  0.147 
0.335  ±  0.141 
0.317  ±  0.126 


Table  48.  Fishman’s  Number  and  Size  of  Batches  for  AR(2)  with  Measurement  Noise  Model 


Transient 

Type 

Number  of  Batches 
Am  ^m 

Size 

Ab 

ARMA(1,1)  Data  with  = 

—0.8  and  6i  —  0.3 

None 

90.69 

121.40 

23.92 

16.74 

2.0 

Step 

96.60 

145.20 

32.11 

24.15 

2.0 

Ramp 

238.32 

253.60 

58.63 

62.84 

2.0 

Exp 

307.29 

237.96 

21.75 

27.34 

2.0 

Sine 

174.48 

214.16 

26.73 

20.71 

1.5 

Step 

137.29 

192.70 

34.69 

26.71 

1.5 

Ramp 

322.00 

255.01 

38.84 

57.14 

1.5 

Exp 

410.30 

206.82 

9.26 

22.45 

1.5 

Sine 

135.80 

185.58 

27.30 

18.64 

1.0 

Step 

274.15 

239.78 

24.47 

27.87 

1.0 

Ramp 

385.35 

236.77 

17.30 

38.13 

1.0 

Exp 

398.26 

213.89 

8.32 

21.59 

1.0 

Sine 

111.08 

166.99 

35.51 

25.97 

0.5 

Step 

478.40 

186.38 

4.38 

6.58 

0.5 

Ramp 

433.29 

227.19 

6.32 

14.72 

0.5 

Exp 

389.66 

244.51 

7.57 

13.80 

0.5 

Sine 

143.93 

206.44 

32.13 

24.42 

ARMA(1,1)  Data  with  = 

—0.8  and  8i  =  —0.3 

None 

445.35 

117.92 

2.76 

2.45 

2.0 

Step 

344.62 

206.18 

14.16 

22.41 

2.0 

Ramp 

406.93 

218.30 

27.53 

51.60 

2.0 

Exp 

433.84 

184.01 

16.68 

40.21 

2.0 

Sine 

304.61 

250.19 

20.06 

24.34 

1.5 

Step 

356.33 

205.22 

15.04 

25.89 

1.5 

Ramp 

459.19 

190.77 

17.07 

41.27 

1.5 

Exp 

490.26 

163.07 

10.64 

31.62 

1.5 

Sine 

335.32 

261.54 

20.82 

27.09 

1.0 

Step 

437.35 

153.90 

8.25 

22.77 

1.0 

Ramp 

509.72 

170.01 

10.79 

32.45 

1.0 

Exp 

545.61 

134.65 

5.01 

18.48 

1.0 

Sine 

389.81 

273.28 

18.36 

26.69 

0.5 

Step 

537.70 

140.19 

3.62 

12.71 

0.5 

Ramp 

612.63 

122.00 

2.95 

9.62 

0.5 

Exp 

553.72 

182.83 

9.57 

29.77 

0.5 

Sine 

459.79 

269.77 

12.75 

19.98 
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Table  49. 


Fishman’s  Number  and  Size  of  Batches  for  AR(2)  with  Measurement  Noise  Model 


Transient 

Number  of  Batches 

Size 

Any{ay)  Type 

Am 

Ab 

ARMA(1,1)  Data  with  <i>i  = 

—0.3  and  6i  =  0.8 

None 

24.50 

11.57 

48.28 

18.11 

2.0 

Step 

366.52 

212.91 

12.89 

20.88 

2.0 

Ramp 

315.95 

229.52 

38.00 

59.42 

2.0 

Exp 

358.33 

183.50 

8.70 

17.56 

2.0 

Sine 

68.30 

183.98 

43.91 

23.37 

1.5 

Step 

353.73 

210.38 

15.26 

24.89 

1.5 

Ramp 

310.38 

238.02 

35.65 

54.95 

1.5 

Exp 

314.34 

169.43 

6.14 

14.60 

1.5 

Sine 

386.59 

381.52 

25.54 

29.87 

1.0 

Step 

422.17 

185.46 

6.03 

13.71 

1.0 

Ramp 

138.28 

197.03 

76.45 

61.65 

1.0 

Exp 

194.43 

195.40 

33.19 

51.17 

1.0 

Sine 

74.44 

165.40 

46.06 

22.38 

0.5 

Step 

237.27 

91.12 

8.80 

14.75 

0.5 

Ramp 

113.16 

89.64 

55.10 

63.32 

0.5 

Exp 

324.57 

253.90 

13.95 

32.21 

0.5 

Sine 

107.26 

209.44 

43.03 

22.45 

ARMA(1,1)  Data  with  (pi  = 

—0.3  and  6i  =  —0.8 

None 

392.19 

140.37 

3.23 

2.52 

2.0 

Step 

367.54 

156.43 

5.53 

11.69 

2.0 

Ramp 

360.92 

185.44 

9.15 

23.06 

2.0 

Exp 

332.24 

189.78 

12.80 

29.37 

2.0 

Sine 

308.30 

209.98 

11.59 

17.91 

1.5 

Step 

360.68 

158.45 

5.54 

12.74 

1.5 

Ramp 

345.85 

193.91 

12.43 

29.67 

1.5 

Exp 

341.26 

199.93 

12.69 

29.50 

1.5 

Sine 

325.56 

217.58 

10.78 

15.93 

1.0 

Step 

343.03 

170.85 

9.70 

25.60 

1.0 

Ramp 

358.47 

202.22 

11.41 

27.81 

1.0 

Exp 

374.15 

203.48 

9.36 

22.79 

1.0 

Sine 

343.66 

223.21 

9.92 

14.76 

mm 

Step 

384.52 

190.47 

9.06 

24.39 

Ramp 

397.15 

214.06 

8.77 

21.50 

Exp 

412.07 

206.51 

6.83 

15.81 

Sine 

385.97 

227.19 

8.28 

13.56 

Table  50.  Fishman’s  Number  and  Size  of  Batches  for  AR(2)  with  Measurement  Noise  Model 


Transient  Number  of  Batches 
Afj.y{ay)  Type 


ARMA(1,1)  Data  with  =  0.3  and  6i  =  0.8 


None 

2.0 

Step 

2.0 

Ramp 

2.0 

Exp 

2.0 

Sine 

1.5 

Step 

1.5 

Ramp 

1.5 

Exp 

1.5 

Sine 

Step 

Ramp 

Exp 

Sine 


Step 

Ramp 

Exp 

Sine 


37.77 


666.5 

434.0 

429.1 

238.4 


695.71 

442.89 

411.77 

426.70 


591.16 

303.04 

314.71 

514.44 


242.5 
147.2 
314.87 
363.92 


19.16 


399.02 

388.28 

335.44 

541.72 


453.60 
261.93 
373.03 
4P^  38 


39 

25 


372.34 

434.28 


Size 

Ab 

=  0.3  and  6i  =  0.8 

33.92 

17.63 

6.16 

12.44 

24.75 

45.19 

8.37 

16.26 

31.83 

32.00 

6.59 

14.33 

16.91 

37.54 

10.43 

23.49 

27.98 

30.15 

12.54 

22.39 

25.03 

44.26 

29.09 

48.94 

20.13 

26.01 

11.71 

19.31 

59.17 

63.17 

35.96 

55.82 

24.09 

25.39 

ARMA(1,1)  Data  with  <f>i  =  0.3  and  6i  =  —0.8 


142.83 


141.27 

136.71 

136.97 

125.21 


146.11 

130.87 

125.44 

121.15 


63.41 


63.18 

69.59 

66.65 

67.22 


63.60 

66.88 

70.52 

67.49 


8.87 


8.61 

11.26 

10.11 

12.43 


8.27 

12.86 

15.14 

13.81 


5.90 


6.30 

11.66 

9.85 

12.07 


5.47 

16.96 

21.36 

13.33 


142.13 

64.53 

8.83 

7.02 

L29.61 

73.43 

16.36 

24.96 

L24.21 

71.78 

18.04 

26.97 

120.67 

67.65 

14.55 

14.03 

12". 29 

71.98 

16,78 

26.35 

L26.95 

75.69 

21.63 

33.44 

L39.n 

73.22 

14.09 

19.27 

L39.16 

72.51 

12.76 

11.73 
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Table  51. 


Fishman’s  Number  and  Size  of  Batches  for  AR(2)  with  Measurement  Noise  Model 


Transient 
A/Xj,(a„)  Type 

Number  of  Batches 

Size 

As 

ARMA(1,1)  Data  with  (f>i  = 

—0.8  and  0^  =  0.3 

None 

50.89 

18.10 

23.39 

12.54 

WESM 

Step 

50.68 

18.47 

22.82 

Ramp 

49.75 

19.07 

24.63 

13.52 

Exp 

50.69 

18.76 

23.43 

13.58 

■■ 

Sine 

48.56 

20.69 

16.34 

1.5 

Step 

51.58 

22.66 

12.56 

1.5 

Ramp 

48.55 

26.26 

15.30 

1.5 

Exp 

47.21 

26.53 

15.83 

1.5 

Sine 

41.01 

32.92 

18.79 

1.0 

Step 

50.68 

19.35 

23.27 

12.99 

1.0 

Ramp 

45.01 

20.85 

29.87 

18.18 

1.0 

Exp 

39.93 

21.57 

37.75 

26.96 

1.0 

Sine 

42.28 

18.77 

32.54 

18.32 

0.5 

Step 

49.88 

.18.77 

24.17 

0.5 

Ramp 

37.63 

23.93 

50.21 

0.5 

Exp 

39.09 

23.19 

47.43 

|{3Ba 

0.5 

Sine 

45.40 

20.48 

33.14 

20.11 

ARMA(1,1)  Data  with  <f>i  = 

0.8  and  6i  =  —0.3 

None 

49.29 

24.44 

13.37 

2.0 

Ramp 

48.12 

17.85 

23.97 

13.41 

2.0 

Exp 

48.99 

18.29 

22.94 

13.10 

2.0 

Sine 

48.18 

24.06 

14.15 

1.5 

Step 

49.05 

17.27 

22.68 

12.41 

1.5 

Ramp 

46.48 

19.65 

26.51 

16.93 

1.5 

Exp 

48.32 

17.86 

23.45 

13.90 

1.5 

Sine 

46.40 

25.68 

15.44 

Step 

47.99 

17.97 

23.50 

13.17 

III 

Ramp 

44.86 

19.98 

28.48 

18.81 

III 

Exp 

42.77 

20.36 

31.03 

21.98 

Sine 

43.03 

17.88 

29.08 

16.59 

0.5 

Step 

47.92 

18.71 

23.95 

15.75 

Ramp 

22.33 

40.37 

34.86 

Exp 

41.88 

36.90 

■■ 

Sine 

30.69 

19.52 
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gorithm  does  not  perform  as  well  on  sequences  that  have  slow  shifts  or  quick  large  shifts 
from  the  transient  to  steady  state.  One  possible  explanation  for  this  perfomance  may  be 
that  the  scalar  k  is  consistently  estimated  to  be  greater  than  0.7.  As  the  scalar  k  nears  one, 
the  first  element  of  the  Kalman  filter  gain  also  nears  one.  The  closer  the  scalar  k  is  to  1.0, 
the  more  the  Kalman  filter  disregards  the  model  state  estimate  and  uses  the  last  data  for 
the  best  state  estimate.  If  the  scalar  k  actually  equals  one,  the  predicted  estimate  is  the 
last  observations  propagated  by  the  state  transtition  matrix.  The  result  of  a  large  Kalman 
filter  gain  is  that  the  state  estimates  quickly  adjust  to  changes  in  the  measurement  sequence. 
With  a  high  Kalman  filter  gain,  a  jump  discontinuity  in  the  sequence  is  only  apparent  in  the 
residual  sequence  as  a  single  residual  at  the  time  of  the  discontinuity  and  of  the  magnitude 
of  the  jump  shift.  For  this  application,  the  filter’s  adjustment  hides  the  effects  of  processing 
transient  data.  When  transient  data  is  processed,  the  filter  adjusts  because  of  the  high  gain, 
and  the  residuals  remain  relatively  small.  Without  a  significant  change  in  the  residuals,  the 
cusum  algorithm  has  no  means  to  detect  a  change. 

When  applied  to  output  of  waiting  times  from  a  simulation  of  an  M/M/1  queue,  k  is 
estimated  to  be  1.0.  Therefore,  the  algorithm  adjusts  to  every  shift  in  the  output  sequences. 
The  residuals  are  just  the  difference  between  sequential  values  of  the  original  output  sequence. 
Thus,  for  that  sample  of  simulation  output,  the  residual  monitoring  algorithm  is  unsuccessful. 
One  proposed  solution  is  to  run  two  filters,  one  with  the  estimated  gain  and  another  filter 
with  a  gain  of  zero.  Perhaps  by  monitoring  the  rate  of  divergence  of  the  filter  with  zero  gain, 
the  transient  can  be  identified.  Another  potential  approach  may  be  to  employ  a  bank  of 
filters,  like  the  MMAE  parameter  estimation  technique,  around  the  estimated  steady-state 
mean.  By  monitoring  the  a  posteriori  probability  of  the  filters’  parameters,  perhaps  a  good 
truncation  point  can  be  determined.  This  approach  is  developed  and  tested  in  the  next 
chapter. 
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^.7  Summary 

Data  truncation  is  a  commonly  accepted  method  of  dealing  with  initialization  bias  in 
discrete-event  simulation.  An  algorithm  for  determining  the  appropriate  initial-data  trunca¬ 
tion  point  for  univariate  output  is  studied.  The  technique  entails  beginning  at  the  end  of  the 
simulation  output  and  estimating  a  steady-state  output  model  in  a  Kalman  filter  framework. 
Using  the  estimated  model,  the  Kalman  filter  is  applied  and  the  residuals  are  monitored  us¬ 
ing  a  cumulative  sums  (cusum)  quality  control  algorithm.  The  estimated  truncation  point 
is  selected  when  the  cusum  algorithm  indicates  the  Kalman  filter  residual  sequence  is  “out- 
of-control”  . 

A  Monte  Carlo  analysis  using  data  sets  generated  from  AR(2)  and  ARMA(1,1)  with 
induced  transients  is  used  to  evaluate  the  technique.  The  performance  measures  include  the 
ability  to  construct  reliable  confidence  intervcils  for  the  mean  response  and  passing  Schruben’s 
test  for  no  initialization  bias.  The  test  cases  selected  probably  have  significantly  more  vari¬ 
ability  than  actual  discrete-event  simulation  output.  Reasonable  results  are  obtained  for 
sequences  with  positive  autocorrelations  at  lag  one.  Obviously,  detecting  small  changes  in 
the  process  mean  is  difficult. 

The  residual  monitoring  algorithm  does  not  preform  well  in  a  limited  test  on  actual 
simulation  output.  The  simulation  output  is  of  waiting  times  for  an  M/M/1  queue.  The 
parameter  estimation  step  results  in  k  fo  1.0.  Thus  the  resulting  Kalman  filter  adapted 
to  every  change  in  the  output  sequence  with  no  significant  pattern  in  the  residuals.  The 
technique  is  unable  to  detect  the  transient  observations.  The  next  chapter  explores  a  different 
approach  for  the  univariate  single-run  steady-state  identification  problem. 
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V.  Single-Run  Steady -State  Identification:  Multiple  Model  Adaptive 

Estimation  (MMAE)  Approach 


5. 1  Introduction 

The  truncation-point  identification  approach  proposed  in  this  chapter  is  based  on  Mul¬ 
tiple  Model  Adaptive  Estimation  (MMAE).  After  estimating  the  steady-state  Kalman  filter 
parameters,  the  MMAE  technique  is  applied  to  calculate  a  time- varying  estimate  of  the  out¬ 
put  mean.  A  truncation  point  is  selected  when  the  MMAE  mean  estimate  is  “near”  the 
steady-state  meain  estimate. 

In  the  MMAE  technique,  the  data  sequence  is  processed  through  a  bank  of  Kalman  fil¬ 
ters,  each  with  a  different  set  of  parameters.  After  processing  an  observation,  the  probability 
of  each  filter  having  the  correct  parameters  is  calculated  based  on  the  filter  residuals.  The 
MMAE  time- varying  estimate  of  the  parameters  is  calculated  with  a  probabilistic  weighted 
average  of  the  filter  parameters.  MMAE  is  introduced  and  discussed  in  Section  3.4.2. 

In  this  application,  the  steady-state  Kalman  filter  parameters  are  estimated  using 
the  correlation  technique  [66].  The  filters  in  the  MMAE  bank  share  the  same  steady-state 
parameter  estimates,  except  they  have  different  mean  estimates.  Starting  at  the  beginning 
of  the  sequence,  the  data  sequence  is  processed  forward  through  the  MMAE  bank  of  filters. 
After  processing  each  observation,  the  MMAE  filter  probabilities  are  updated  and  a  new 
MMAE  mean  estimate  based  on  all  the  previous  data  is  calculated.  Effective  steady-state 
output  may  be  detected  in  two  ways.  First,  the  filter  probability  for  the  filter  using  the 
steady-state  mean  estimate  should  be  near  one.  Second,  when  the  MMAE  mean  estimate  is 
within  a  small  tolerance  of  the  steady-state  mean  estimate  for  a  predetermined  number  of 
consecutive  observations,  the  truncation  point  can  be  selected. 

In  related  research,  Howard  [37]  uses  MMAE  to  construct  confidence  intervals  for  the 
univariate  output  of  discrete-event  simulations.  He  tests  three  different  confidence  interval 
techniques.  The  first  technique  is  based  on  the  estimated  steady-state  parameters  and  vari- 
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ances.  The  second  confidence  interval  technique  uses  the  MMAE  estimate  of  the  variance  for 
the  mean  estimator,  and  the  final  technique  is  based  upon  the  final  MMAE  filter  probabili¬ 
ties.  Since  Howard  uses  the  same  Kalman  filter  model  formulation,  this  analysis  of  MMAE 
characteristics  provides  some  insight  to  Howard’s  confidence  interval  results. 

Although  the  same  Kalman  filter  formulation  as  used  in  Chapter  IV  is  applied,  a 
different  parameter  estimation  scheme  is  used.  After  discussing  parameter  estimation,  the 
MMAE  truncation  approach  is  developed,  and  a  Monte  Carlo  analysis  follows.  This  chapter 
is  divided  into  the  following  major  sections:  parameter  estimation,  MMAE  implementation, 
and  Monte  Carlo  Analysis. 

5.2  Parameter  Estimation 

In  this  section,  the  model  formulation  is  reviewed.  The  parameter  estimation  with  the 
correlation  technique  is  developed.  The  accuracy  of  the  variance  estimates  is  analyzed. 

In  this  chapter,  the  AR(2)  with  measurement  noise  formulation,  from  Section  4.2.1,  is 
applied.  Using  the  same  system  matrices  shown  in  Equation  (45),  the  simulation  output  t/„ 
is  modeled  as  the  sum  of  a  constant  mean,  AR(2)  correlated  noise,  and  independent  white 
measurement  noise.  The  four  parameters  that  must  be  estimated  are  the  steady-state  mean 
fiy,  the  autoregressive  coefficients  and  <^2,  a^nd  the  first  element  of  the  Kalman  filter  gain 
k.  Based  on  estimates  of  these  parameters  and  the  sample  residual  mean,  the  dynamic  noise 
variance  Qd,  measurement  noise  variance  R,  and  the  Kalman  filter  state  estimate  variances 
P~  and  P'^  can  be  estimated,  if  desired. 

In  Chapter  IV,  the  four  unknown  parameters  are  estimated  using  least  squares  esti¬ 
mation.  The  data  is  processed  from  the  end  of  the  simulation  output  until  the  least  squares 
estimates  pass  a  heuristic  criterion.  In  contrast,  this  approach,  like  Schruben’s  [87],  assumes 
that  the  last  half  of  the  data  sequence  has  effectively  attained  a  stationary  steady-state 
output  distribution.  Therefore,  using  the  fixed  sample  size  of  half  the  number  of  data,  the 
steady-state  parameters  are  estimated. 
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Using  Mehta’s  [66]  correlation  technique  for  parameter  estimation,  the  first  three  the¬ 
oretical  lagged  autocorrelations  for  an  AR(2)  with  measurement  noise  model  are  equated 
with  the  sample  autocorrelations.  In  this  formulation,  the  AR(2)  noise  term  ^(t„)  equals  the 
first  element  of  the  state  estimate,  Xi(f~)  =  ^(tn)-  The  state  vector  x(t„)  is  a  random  vector 
with  mean  of  x(tn)  and  variance  of  P".  Since  the  measurement  noises  are  independent  of 
the  state  estimates  and  independent  in  time,  the  lagged  simulation  output  autocovariances 
tire  the  same  as  for  the  AR(2)  dynamics  state  estimate: 

7v(l)  =  7^(1)  =  t^7.-(0) 

7v(2)  =  7^7(2)  =  <^17.5(1)  +  Mij(O)  =  r^7^(0)  +  <^27*5(0) 

7v(3)  =  7»i(3)  =  <^i7ri(2)  +  hliiil) 

The  variance  7,j(0)  equals  the  variance  of  the  first  element  of  the  state  estimate  P{[.  The 
variance  of  the  simulation  output,  shown  in  Equation  (55),  can  be  expressed  as  the  variance  of 
the  first  element  of  the  state  estimate  plus  the  measurement  noise  variance,  7y(0)  =  P{,  +  R. 
The  ratio  or  equivalently  ,  is  seen  to  equal  the  unknown  scalar  k  by  Equation  (48) 
on  page  91.  Dividing  the  autocovariances  by  the  variance  7y(0)  results  in  the  theoretical 
autocorrelations  Py{i)-  After  substituting  in  the  scalar  k,  the  autocorrelation  equations  are 

Pv(l)  ~ 

Py{2)  =  <^i/Oy(l)  -I-  hk  (62) 

py(3)  =  4>ipy{2)  4-  4>2py{^) 

Using  these  relationships,  a  quadratic  equation  for  <f>i  can  be  derived.  Solving  the  first  two 
autocorrelations  in  Equation  (62)  for  k  results  in 

^  ^  Pv(l)  -  4>2py{\)  ^  Py{2)  -  (i>lpy{l) 

4>2 
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Cross  multiplying  and  substituting  in  the  relationship  from  the  third  autocorrelation, 

(f>2  =  results  in 


=  />.(!)  - 

=  ,>,(3)  -  :^ip,(2)  - 


All  the  terms  are  moved  to  one  side  of  the  equation  eind  grouped  terms  in  powers  of 


PyC^)  j-2  ,  2pv(2)Pv(3) 


.  /I\i2  ry\-/  J^2  .  -rv\-jrv\ 


—  2py{2)(f>i  +  p„(3)  — 


pM 

A>y(l) 


=  0 


The  actual  parameter  estimates  for  <j>i,  (j>2,  and  k  are  calculated  using  the  sample 
autocorrelations.  Using  the  last  half  of  the  output  sequence,  the  steady-state  output  mean 
Py  is  estimated  with  the  sample  mean. 


N 


y  = 


N-2 


£  Vn 


The  sample  autocorrelations  ry(i)  at  lag  i  are  calculated  with 


E„_If.Jyn-y)(yr,H-y) 

r,(i)  =  - 


E"  -  sr 


(63) 


(64) 


Substituting  the  sample  autocorrelations,  a  quadratic  equation  in  terms  of  <^i  is  deter¬ 
mined; 


'-.(1)  -  +  f  -  2’'.(2)')  h  +  fr,{3)  - 


^y(l) 


^y(l). 


=  0 


(65) 


Using  the  solutions  for  a  quadratic  equation,  two  estimates  of  4>i  may  be  available.  For  each 
quadratic  solution  for  (p\ ,  the  third  autocorrelation  equation  is  used  to  estimate  the  second 
autoregressive  coefficient,  <^2  =  effort  to  minimize  the  numerical  error,  the 
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Table  52.  Correlation  Technique  Parameter  Estimation  Summary 


Mv  y 

Av 

10.00  10.00  0.32 

0.60  0.63  0.12 

0.30  0.27  0.11 

0.57  0.80  0.05 

pair  of  autoregressive  parameters  (^i,  ^2)  which  minimizes  |7‘v(3)— ()iir„(2)— ^2’"y(l)|  from  the 
third  autocorrelation  equation  is  selected.  (In  a  test  of  1000  runs,  the  truncation  algorithm 
using  the  other  pair  of  estimates  obtained  very  similar,  but  slightly  inferior,  results.)  Using 
these  estimates  for  the  autoregressive  parameters,  k  —  from  the  autocorrelation 

equation  for  lag  one.  The  resulting  estimates  of  (l>i,  ^2  and  k  are  the  correlation  technique 
estimates. 

Based  on  the  estimates,  the  gain  and  variances  can  be  determined.  Using  Equation  (51) 
the  Kalman  filter  gain  K  is  calculated.  The  estimation  of  the  noise  and  state  variances 
{Qd,  R,  and  P"*")  is  the  same  as  described  in  Section  4.3.3.  However,  the  bias  in  the 

estimation  of  k  affects  the  variance  estimates,  as  demonstrated  below. 

As  a  test  of  this  estimation  scheme,  this  correlation  technique  is  applied  to  3,000 
generated  data  sequences.  The  model  for  the  generated  data  is  an  AR(2)  with  measurement 
noise  with  autoregressive  coefficients  of  =  0.6  and  (f>2  =  0.3,  dynamics  noise  variance 
Qd  =  1.0,  and  measurement  noise  variance  /?  =  1.0.  For  each  generated  sequence,  the 
parameter  estimates  eire  based  on  1,000  steady  state  observations.  As  seen  in  Table  52,  the 
estimates  of  the  mean  and  the  autoregressive  coefficients  are  fairly  accurate;  however,  the 
estimate  of  the  scalar  k  is  significantly  in  error.  Since  the  value  of  the  estimate  k  is  integral 
to  the  estimation  of  the  noise  variances  (See  Section  4.3.3),  these  calculations  are  completed 
twice;  once  with  the  estimated  scalar  k  and  again  with  the  true  k.  Table  53  shows  that 
the  biased  estimate  k  induces  significant  error  into  the  noise  variance  estimates  Qj  and  R. 
However,  if  k  can  be  accurately  estimated,  the  noise  variance  estimates  are  also  accurate. 

Most  estimation  schemes  for  Kalman  filter  parameters,  such  as  MLE,  MMAE,  and 
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Table  53.  Variance  Estimation  Summary 


1.00  1.68  0.18 

1.00  0.48  0.13 

2.27  2.80  0.39 

True  k 

1.00  1.01  0.08 

1.00  1.00  0.05 

2.27  2.33  0.27 

Table  54.  Sum  of  Squared  Residuals  for  Veirious  Scalar  k 


■ 

Estimated 

Ayi  <i>2 

True 

n 

%  Change 

EnJTrl  %  Change 

104,472.8 

124.6 

104,729.6 

125.2 

65,545.0 

40.9 

64,423.2 

38.5 

54,414.0 

17.0 

53,718.6 

15.5 

lilSl 

49,763.7 

7.0 

49,355.6 

6.1 

0.4 

47,620.5 

2.4 

47,397.2 

1.9 

0.5 

46,711.8 

0.4 

46,615.6 

0.2 

1.6 

46,506.4 

0.0 

46,502.8 

0.0 

0.7 

46,744.9 

0.5 

46,812.8 

0.7 

0.8 

47,286.4 

1.7 

47,411.9 

2.0 

0.9 

48,047.6 

3.3 

48,221.6 

3.7 

1.0 

48,978.6 

5.3 

49,194.9 

5.8 

Note: 

True  k 

=  0.57. 

least  squares,  are  based  on  functions  of  the  squared  residuals.  In  an  effort  to  assess  the 
ambiguity  in  the  estimation  of  the  scalar  fc,  one  sequence  of  20,000  steady-state  observations 
is  examined.  The  sequence  was  generated  with  an  AR(2)  with  measurement  noise  model. 
The  sum  of  squared  residuals  are  calculated  for  each  value  of  the  scalar  k  vciried  at  increments 
of  0.1  over  its  feasible  range  of  zero  to  one.  The  first  set  of  columns  for  the  sum  of  squared 
residuals  in  Table  54  is  calculated  with  the  parameter  estimates  of  fiy  —  10.15  =  10.0), 

=  0.595  (^1  =  0.6),  and  ^2  —  0.291  {4>2  =  0.3).  Even  with  these  relatively  accurate 
parameter  estimates,  the  effect  of  the  scalar  fc  is  a  very  small  percentage  change  in  the  sum 
of  20,000  residuals  over  most  of  the  feasible  region.  To  ensure  that  the  estimation  errors  of  the 
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other  paxameters  are  not  reducing  the  effect  of  the  variation  in  A,  the  test  is  conducted  again 
using  the  true  pctrameters  except  for  k.  Even  when  fixing  the  parameters  which  generated 
the  data  to  their  true  values,  the  effect  of  varying  k  is  minimal  for  most  of  the  feasible  range. 
The  length  of  this  data  sequence  is  ten  times  greater  than  the  simulation  output  sequences 
tested  in  this  research.  Thus,  the  value  of  k  may  not  be  estimated  accurately  with  a  limited 
number  of  observations  available  in  many  simulation  applications. 

Howard  [37]  in  his  confidence  interval  techniques  uses  a  slightly  different  approach  to 
estimating  the  parameters.  He  uses  the  first  two  correlations  to  estimate  the  autoregressive 
coefficients  assuming  a  value  for  the  scalar  k.  A  linear  search  for  k  between  its  bounds 
of  zero  cind  one  is  applied  to  minimize  the  sum  of  squared  residuals.  Howard  applies  the 
the  same  equations  to  calculate  the  variance  as  the  results  in  Table  53.  In  his  tests,  the 
scalar  k  is  overestimated.  Thus,  the  dynamics  noise  variance  generally  is  overestimated  and 
the  measurement  noise  variance  is  underestimated.  These  biased  variance  estimates  are  the 
basis  of  the  first  confidence  interval  technique  Howard  tested.  Probably  because  of  the  bias 
in  the  estimation  of  k,  the  confidence  interval  technique  does  not  perform  as  well  as  other 
techniques. 

Table  54  indicates  that  the  value  of  k  has  a  minor  impact  on  the  sum  of  squared 
residuals  for  this  very  long  sequence.  The  implication  is  that  k  may  not  be  estimated 
accurately  with  a  limited  number  of  observations.  In  the  opposite  view,  the  value  of  k  does 
not  significantly  degrade  the  performance  of  the  Kalman  filter.  Perhaps  the  value  of  k  may 
be  arbitrarily  set  to  achieve  desired  filter  performance  characteristics. 

Using  the  above  estimation  scheme  based  on  the  correlation  technique,  estimates  of 
the  mean,  the  autoregressive  coefficients,  and  the  scalar  k  are  obtained  with  only  two  passes 
through  the  data  sequence.  Using  these  estimated  parameters,  the  last  half  of  the  data 
sequence  is  processed  through  the  estimated  steady-state  Kalman  filter.  The  mean  residual 
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squaxed  error  is  used  as  an  estimate  of  the  residual  variance: 

HP-h’'  +  «=-|  f;  r;  (66) 

With  these  estimates,  a  bank  of  Kalman  filters  can  be  set  up,  and  MMAE  may  be  applied. 

5.S  Multiple  Model  Adaptive  Estimation  (MMAE)  Implementation 

This  section  describes  the  application  of  MMAE  to  the  problem  of  steady-state  identi¬ 
fication.  The  implementation  issues,  particularly  the  appropriate  magnitude  of  the  scalar  k, 
are  discussed.  The  section  identifies  two  bases  for  determining  that  steady-state  observations 
are  being  processed. 

MMAE  is  introduced  in  Section  3.4.2  as  a  parameter  estimation  technique.  In  MMAE, 
a  bank  of  Kalman  filters,  each  with  a  different  set  of  parameters,  is  run  through  the  data 
sequence.  After  processing  each  observation,  the  probability  of  each  filter  having  the  cor¬ 
rect  parameters  is  calculated  with  Equation  (36).  Using  the  filter  probabilities,  an  MMAE 
estimate  of  the  unknown  parameters  is  determined  with  Equation  (39)  with  conditional 
covariance  matrix  of  the  estimated  parameters  calculated  with  Equation  (40). 

In  this  application,  the  steady-state  system  parameter  estimates,  calculated  with  the 
correlation  technique,  are  used  in  the  MMAE  bank  of  Kalman  filters.  However,  each  filter 
is  based  on  a  different  mean  estimate.  Thus,  the  MMAE  filter  probabilities  and  estimates 
give  indication  to  changes  in  the  mean  of  the  data  sequence. 

Application  of  the  MMAE  approach  involves  determination  of  the  following: 

•  Number  and  spacing  of  parameter  estimates  in  the  MMAE  bank 

•  Value  of  the  Kalman  filter  scalar  k 

•  Filter  prior  probabilities,  pi{to)  in  Equation  (36) 

•  Determining  if  and  when  to  reset  the  filter  probabilities  pi(tn) 
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•  Establishing  rules  to  declare  steady-state  by  selecting  Uq 

As  Howard  [37,  39]  reports,  these  issues  can  be  critical  to  successfully  applying  MMAE 
to  simulation  output.  The  issues  are  difficult  to  resolve  since  their  effects  interact  in  the 
performance  of  the  MMAE  technique.  Each  of  these  issues  is  addressed  in  the  following 
sections. 


5.S.1  Number  and  Spacing  of  Filters.  While  in  some  MMAE  applications  many  fil¬ 
ters  are  incorporated  into  the  bank,  the  filters  in  this  application  are  intended  to  signal  three 
possibilities  about  the  output  sequence:  a  downward  bias,  an  upward  bicis,  or  no  bias.  The 
objective  is  to  truncate  the  initial  biased  data  that  significantly  affect  the  mean  estimator. 
Therefore,  in  this  application,  detecting  biases  of  different  sizes  is  important,  but  not  esti¬ 
mating  the  magnitude  of  a  bias.  The  selected  bank  consists  of  three  filters,  one  centered  on 
the  steady-state  mean  estimate  and  the  others  spaced  above  and  below. 

The  spread  between  the  filter  means  also  influences  the  MMAE  filter  probabilities, 
and  hence  the  MMAE  mean  estimates.  Let  the  steady-state  mean  based  on  the  last  half 
of  the  data  be  represented  by  y.  The  mean  estimates  in  the  three  filters  are  p-j  =  y  +  Sj 
where  Sj  is  negative  for  the  low  filter,  zero  for  the  center  filter,  and  positive  for  the  high  one. 
The  effect  of  spacing  is  examined  with  k  set  equal  to  zero  and  equal  to  one.  These  values 
evaluate  the  possible  range  of  the  filter  gains.  A:  =  0  means  the  Kalman  filter  relies  only 
on  the  dynamics  model  and  disregards  measurement  updates.  In  contrast,  /b  =  1.0  means 
that  the  state  estimates  are  based  on  the  propagation  of  the  last  two  measurements.  In  both 
cases,  the  filter  parameters  need  to  be  spaced  far  enough  apart  to  differentiate  between  the 
filters’  residuals.  The  actual  spacing  necessary  is  a  function  of  the  variation  in  the  simulation 
output  being  processed. 


First,  the  Kalman  filter  residuals  are  examined  with  no  measurement  updates.  If 

0 


A:  =  0,  the  Kalman  filter  gain  K  = 


0 


by  Equation  (51).  Assuming  stationary  estimates 


for  AR(2)  coefficients  and  (f>2,  without  measurement  updates,  the  Kalman  filter  state 
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estimates  x(t„ )  and  x(t^ )  decay  to  zero  vectors.  For  each  filter,  the  measurement  is  the 
data  yn  minus  that  filter’s  mean  For  each  of  three  filters,  the  residual  at  time  is 


=4-Hx(C) 
=  yn-fii 
=  yn-y-^j 


The  data  yn  and  the  estimated  steady-state  mean  y  are  the  same  for  each  filter.  Each  filter’s 
residuals  are  determined  by  the  variation  in  yn  and  the  size  of  6j.  Since  the  relative  magni¬ 
tudes  of  the  filter  residuals  determine  the  MMAE  filter  probabilities,  the  size  of  6j  compared 
to  the  variation  in  may  affect  the  stability  of  the  filter  probabilities.  Relatively  stable 
filter  probabilities  maJte  the  MMAE  mean  estimate  smoothly  progress  as  more  observations 
are  processed.  Therefore  Sj  should  probably  be  large  compared  to  <t„. 

Similarly,  the  MMAE  filter  parameters  should  be  widely  spaced  when  the  Kalman 
filters’  estimates  rely  extensively  on  the  measurements.  For  k  =  \,  the  Kalman  filter  gain 


K  = 


1 

0 


by  Equation  (51).  From  Equations  (22),  (25)  and  (45),  the  corresponding  state 


estimates  for  the  jth  filter  are  Xj(t^_i)  = 
residuals  for  each  of  the  filter  are 


and  x(<„ ) 


^i4-i  + 


‘-n-l 


.  The 


=2^-Hxj(0 

=  ^n-  -  hZn-2 

=  (l/n  -  ^ll)  -  Myn-1  -  H)  -  Myn-2  “  M^) 

=  (z/n  -  y  -  Sj)  -  0i(j/„_i  -  y-  6j)  -  ^2{yn-2  -  y  ~  Sj) 

=  (yn  -  y)  -  <^i(yn-i  -  y)  -  Myn-2  -  y)  -  (1  -  -  MS, 

The  last  three  observations  minus  the  steady-state  mean  estimate  are  the  same  for  each 
filter.  For  stationary  AR(2)  processes,  the  factor  (1  —  (^i  —  M  is  bounded  between  0  and  4. 
With  fc  =  1,  the  spacing  between  filters  Sj  must  be  sufficiently  large  to  distinguish  between 
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the  different  filters’  residuals. 

Whether  it  =  0  or  ifc  =  1,  the  differences  between  the  filter  means  need  to  be  sufficiently 
large  to  result  in  signihcant  difference  in  the  filter  residuals,  and  hence,  the  MMAE  filter 
probabilities.  If  the  filter  mecins  are  too  close  together,  the  variation  in  the  output  sequence 
causes  the  probability  to  fluctuate  between  Alters.  If  the  filters  axe  spread  too  far  apart, 
regardless  of  the  transient,  the  significant  probability  remains  with  the  center  filter. 

In  a  Monte  Carlo  analysis,  two  rules  for  determining  the  filter  spread  are  considered 
Initially,  the  fllt‘“rs  are  spread  based  on  the  variation  in  the  filter  residuals.  Using  the 
estimated  steady-state  parameters  and  the  last  half  of  the  output  sequence,  the  residual 
mean  square  error  is  calculated,  HP~H^  +  R  =  En=^i+1  ^n-  The  filter  spread  of  8j  = 

3[HP-H^  -t-  .R]  2  is  vested  initially.  However,  the  Alter  parameter  range  that  worked  for  the 
generated  time-series  models  is  not  adequate  with  simulation  output  of  an  M/M/1  queue. 
The  magnitude  of  the  residuals  can  vary  depending  on  how  well  an  AR(2)  with  measurement 
noise  model  can  fit  the  output  sequence.  Since  the  different  filter  means  are  used  to  detect 
changes  in  the  output  sequence,  the  variation  of  the  output,  rather  than  the  residuals,  seems 
to  provide  a  better  indication  of  the  appropriate  filter  spacing.  Since  no  reliable  variance 
estimate  for  the  output  sequence,  is  readily  available,  the  filters  are  spread  based  on  the 
minimum  and  maximum  data  values  in  the  last  half  of  the  output  sequence.  The  lower  and 
upper  filters  are  positioned  at  90  percent  of  the  distance  between  the  estimated  mean  and 
the  respective  extreme  data  values. 

In  this  subsection,  the  number  of  filters  for  the  MMAE  bank  is  choosen  to  be  three.  The 
filters’  means  are  based  on  the  estimated  mean  and  either  the  residual  standard  deviation  or 
the  extreme  simulation  output  values.  The  next  subsection  investigates  the  effect  of  varying 
the  magnitude  of  the  Kalman  filter  gain. 

5.3.2  Kalman  Filter  Gain.  In  this  subsection,  the  effect  on  the  MMAE  estimates 
of  the  Kalman  Alter  gains  of  different  magnitudes  are  examined.  The  investigation  has 
two  purposes.  The  first  is  to  select  the  value  of  «,  which  deterrplnes  the  magnitude  of  the 


173 


Figure  13.  Sample  AR(2)  with  Measurement  Noise  Data  Sequence 


Kalman  filter  gain,  that  results  in  good  performance  of  the  MMAE  technique.  The  second 
objective  is  to  identify  MMAE  estimates  that  with  an  associated  criteria  may  constitute  a 
good  truncation-point  selection  algorithm.  The  MMAE  filter  probabilities,  state  estimates, 
residuals,  MMAE  mean  estimates  and  variances  are  considered.  The  MMAE  filter  probabil¬ 
ities  and  MMAE  mean  estimates  are  selected  as  potential  indicators  for  a  truncation-point 
selection  algorithm. 

In  investigating  the  effects  of  various  Kalman  filter  gains,  one  data  sequence  is  used 
extensively.  The  sequence,  shown  in  Figure  13,  was  generated  from  an  AR(2)  with  mea¬ 
surement  noise  model  using  the  following  parameters;  autoregressive  coefficients  <f>i  =  0.6, 
<f>2  =  0.3,  dynamics  noise  variance  =  1.0  and  measurement  noise  variance  R  =  1.0.  The 
sequence  consisted  of  2,000  observations.  The  last  1500  observations  are  steady-state  values 
with  a  mean  of  10.0  and  a  standard  deviation  ay  =  2.37.  The  first  500  observations  have  a 
transient  mean  of  fiy  —  2ay  =  5.46.  The  transition  from  the  transient  to  steady-state  is  a  step 
change  in  the  mean  at  the  500th  observation.  While  this  is  an  artificial  transient,  the  large 
step  change  should  be  clearly  identifiable  by  potential  measures  for  selecting  the  appropriate 


174 


Figure  14.  MMAE  Filter  Probabilities  with  fc  =  1.0 


Observations 


truncation  point.  In  fact,  the  transient  is  obvious  in  Figure  13,  whereas  transients  of  actual 
simulation  output  often  are  not  visibly  discernible. 

Applying  the  paurameter  estimation  scheme  to  this  sequence  results  in  the  following 
estimates:  y  =  11.17,  =  0.88,  <^2  =  0.04,  k  —  0.65,  and  the  standard  deviation  of  the 

residuals  is  [HP“H  +  R]i  =  1.55.  In  these  initial  tests,  the  filter  spacing  is  based  on  plus 
and  minus  three  times  the  standard  deviation  of  the  steady-state  filter  residuals.  The  three 
MMAE  filter  means  are  6.52,  11.17,  and  15.82. 

After  applying  MMAE,  the  filter  probabilities  are  examined.  Figure  14  shows  the 
a  posteriori  probabilities  of  the  filter  parameters.  One  potential  alternative  for  steady-state 
identification  is  to  declare  a  truncation  point  when  the  probability  of  the  filter  based  on  the 
mean  estimate  y  achieves  a  high  value.  With  k  =  1.0,  the  filter  probabilities  smoothly  and 
quickly  transitioned  after  the  sequence  changes  to  steady  state  at  the  500th  observation. 

In  Figure  15,  the  residuals  for  the  center  filter,  which  used  the  steady-state  mean 
estimate  y,  are  shown.  The  step  change  in  the  mean  at  observation  500  is  apparent  as  a  single 
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Figure  15.  Residuals  of  Mean  Filter  with  fc  =  1.0 
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large  residual.  Only  one  large  residual  resulted  because  with  k  =  1.0,  the  state  estimates 
adapted  in  one  iteration  to  changes  in  the  data  sequence.  The  one  large  residual  occurs  only 
for  transients  that  end  with  a  step  change  to  steady  state.  Transients  that  decay  gradually, 
which  is  common  in  discrete-event  simulation  output,  produce  no  significant  pattern  in 
the  residuals  because  the  state  estimates  adapt  to  every  small  change  in  the  transient.  In 
Chapter  IV,  the  residuals  are  monitored  in  an  attempt  to  detect  the  change  between  the 
transient  and  steady-state.  Because  the  first  element  of  the  estimated  Kalman  filter  gain 
k  is  generally  very  near  one,  the  state  estimates  adapt  rapidly  to  any  change  in  the  data. 
Therefore,  the  residual  sequence  have  little  or  no  pattern  that  can  be  detected.  Even  with 
smaller  gains,  which  are  attempted  later,  the  residuals  contain  no  easily  identifiable  pattern 
to  detect  a  gradual  change.  Therefore,  tracking  the  residuals  is  not  considered  further  as  a 
viable  basis  for  a  truncation-point  selection  approach. 

The  state  estimates  for  a  particular  filter,  do  indicate  a  change  in  the  output 

sequence.  The  state  estimates  for  this  sequence  are  shown  in  Figure  16.  However,  the 

sequence  of  state  estimates  is  correlated.  In  fact,  with  k  —  1.0,  the  state  estimates  are  simply 
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Figure  16.  Hx  (tn)  of  Mean  Filter  with  fc  =  1.0 
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the  original  sequence  corrected  for  the  filter’s  mean  estimate.  With  lower  values  of  k,  the  state 
estimates  decrease  in  correlation,  but  the  transient  also  becomes  less  obvious.  Because  of  the 
correlation,  detecting  the  change  in  state  estimates  is  no  easier  than  finding  the  chcinge  in 
the  original  data  sequence.  The  MMAE  state  estimates,  determined  by  a  weighted  average 
of  the  filter  state  estimates  and  their  respective  probabilities,  has  no  discernible  pattern. 
Therefore,  the  MMAE  state  estimates  eire  not  pursued  further  as  a  basis  for  identifying 
steady  state. 

Besides  the  individual  filter  state  estimates  and  residuals,  the  MMAE  technique  also 
provides  an  MMAE  mean  estimator  and  variance  of  the  mean  estimator.  Figure  17  shows  the 
MMAE  mean  estimates  fiyitn)^  which  are  the  filter  meams  weighted  by  the  filter  probabilities. 
With  ^  =  1.0  and  the  bank  of  three  filters  as  specified,  the  MMAE  mean  quickly  detects  a 
large  step  change  in  the  mean.  The  assumed  steady-state  mean  estimate  y  is  11.1,  so  the 
center  filter  is  positioned  there.  After  500  trcinsient  observations,  the  MMAE  mean  estimate 
stabilizes  on  the  center  filter’s  mean  estimate.  Because  of  the  clear  indication  of  the 


177 


Figure  17.  MMAE  AvC^n)  with  fc  =  1.0 
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transition  to  steady  state,  the  MMAE  mean  estimator  is  the  basis  of  the  second  truncation- 
point  selection  algorithm  tested  in  a  Monte  Carlo  analysis. 

Figure  18  depicts  the  MMAE  conditional  standard  deviation  of  the  mean  estimate 
calculated  with  Equation  (40)  where  a(t„)  =  Av(*n)-  The  second  confidence  interval 
technique  tested  by  Howard  [37]  is  based  on  the  final  MMAE  variance  estimate.  Because 
of  the  Veiriability  in  these  estimates  as  each  data  point  is  processed,  that  confidence  interval 
technique  may  be  improved  slightly  by  using  an  average  of  a  group  of  estimates  at  the  end 
of  the  data. 

The  MMAE  technique  provides  at  least  two  potential  indicators  of  steady  state.  The 
first  is  that  the  MMAE  filter  probability  for  the  filter  using  the  mean  estimate  of  y  approaches 
one.  The  second  indicator  is  that  the  MMAE  time- varying  mean  estimate  approaches  y.  The 
following  analysis  investigates  the  effect  of  changing  the  magnitude  of  the  Kalman  filter  gain 
on  these  two  indicators. 

Using  the  same  data  sequence,  MMAE  is  applied  with  the  Kalman  filter  gain  scalar 
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Figure  18.  MMAE  with  k  —  1.0 
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Figure  19.  MMAE  Filter  Probabilities  with  k  =  0.5 
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k  set  equal  to  0.5.  Figure  19  shows  the  MMAE  filter  probabilities  for  the  filters.  With  a 
lower  value  for  k,  the  filter  probabilities  are  more  sensitive  to  changes  in  the  data  sequence. 
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Figure  20.  MMAE  fiy{tn)  with  k  =  0.5 


However,  the  probabilities  continue  to  fluctuate  even  when  steady-state  observations  are 
being  processed.  Similar  fluctuations  occur  in  the  MMAE  estimate  of  the  mean 
shown  in  Figure  20. 

The  value  of  ic  selected  determines  the  perfonriance  of  the  MMAE  filter  probabilities 
Pj{tn)  from  Equation  (36).  When  the  sccilar  k  is  set  at  or  near  its  upper  bound  of  one, 
the  MMAE  filter  probabilities  are  very  stable.  As  k  decreases,  the  probabilities,  and  hence 
the  MMAE  mean  estimates  Av(^n))  increase  in  fluctuations.  However,  with  a  lower  k,  the 
probabilities  and  the  MMAE  mean  estimate  respond  faster  to  a  change  in  the  sequence.  By 
varying  k,  a  tradeoff  can  be  made  between  stable  consistent  indicators  and  faster  indicators 
with  more  false  signals.  Because  with  the  higher  k  the  indicators  are  slower  to  respond, 
larger  values  of  k  also  result  in  selecting  truncation  points  that  eliminate  more  of  the  initial 
data.  Since  A:  =  1.0  results  in  clearer  indications  and  conservative  trucation  points,  k  = 
1.0  is  implemented  in  the  Monte  Carlo  analysis  in  this  chapter.  In  Chapter  VI,  the  same 
truncation-point  selection  method  is  applied  to  the  average  of  multiple  replications.  In 
those  applications,  better  truncation  points  are  selected  when  the  estimated  k  is  used  in  the 


algorithm  rather  than  arbitrarily  setting  k  to  one. 

5.S.S  Filter  Probabilities.  To  begin  the  recursive  MMAE  procedure,  filter  a  priori 
probabilities  for  time  to  have  to  be  set.  In  other  applications  of  MMAE,  the  prior  probabilities 
Pj(to)  are  significant  [37].  However,  for  the  test  data  sequence,  the  effect  of  the  priors  is 
insignificant  after  processing  40  observations.  Therefore,  the  priors  are  set  uniformly  with 
each  of  the  three  filters  having  an  a  priori  probability  of  a  third;  Pj(<o)  = 

Each  filter  probability  represents  the  probability  conditioned  on  all  data  available  to 
that  time.  Thus,  the  filter  probabilities  are  affected  by  the  transient  data.  To  eliminate 
the  effect  of  the  transient,  the  probabilities  can  be  periodically  reset  to  their  prior  prob¬ 
abilities.  Resetting  the  filter  probabilities  was  tested,  but  was  found  to  have  little  effect. 
With  dispersed  filter  spacing,  the  probabilities  adapt  rapidly  to  changes  in  the  sequence 
without  resetting.  The  decaying  effect  of  the  previous  data  is  consistent  with  the  decreasing 
significance  of  the  filter  probability  priors  as  more  data  is  processed. 

5.3.4  Declaring  the  Transient:  Heuristic  Rules.  In  order  to  select  the  initial  data 
truncation  point  no,  two  different  heuristics  are  proposed  and  tested.  The  first  approach 
simply  selects  no  at  the  point  when  the  probability  of  the  mean  filter  exceeds  0.9.  The  second 
approach  monitors  the  MMAE  mean  estimate  Py{tn)-  Welch’s  [101]  graphical  truncation- 
point  identification  scheme,  described  in  Section  2.4.2,  estimates  the  time-varying  mean  by 
smoothing  the  average  of  multiple  simulation  replications.  Using  Welch’s  method,  the  ana¬ 
lyst  selects  the  truncation  point  where  the  mean  estimates  stabilize.  In  a  similar  way,  the 
second  proposed  method  selects  the  truncation  point  when  the  MMAE  time-varying  mean 
estimate  Py{tn)  stabilizes  within  a  small  tolerance  of  the  steady-state  mean  estimate,  y. 
When  the  absolute  difference  between  the  mean  estimates,  ABS(/iy(t„)  —  y),  is  less  than  a 
small  tolerance  for  a  few  consecutive  observations,  the  truncation  point  no  is  selected.  For 
these  tests,  the  small  tolerance  is  set  to  1.0  and  the  criterion  is  passed  for  5  consecutive  ob¬ 
servations.  Both  truncation-point  selection  methods  performed  well  for  generated  transients, 
as  seen  in  the  next  section. 


181 


5-4  Monte  Carlo  Results 


A  Monte  Caxlo  analysis  to  test  the  effectiveness  of  using  the  MMAE  approach  to 
determine  data  truncation  points  is  conducted.  Two  types  of  data  are  used.  The  first 
set  of  sequences  are  AR(2)  with  measurement  noise  data.  These  time-series  sequences  use 
the  same  system  parameters  as  the  sample  sequence  in  Figure  13  with  various  types  of 
transients.  The  second  test  set  consists  of  waiting  times  from  simulations  of  an  M/M/1 
queue.  The  truncation-point  selection  methods  perform  well  on  the  time-series  data,  but 
not  as  successfully  on  the  simulation  output.  The  output  from  a  single  simulation  may  be  a 
very  difficult  transient  to  detect. 

5.4-1  AR(S)  with  Measurement  Noise  Data.  The  parameters  used  to  generate  the 
data  are  Hy  —  10.0,  4>i  —  0.6,  <j>2  =  0.3,  Qd  —  10  and  R  —  1.0.  Each  sequence  consists  of  500 
transient  observations  and  1,500  steady-state  observations.  Four  types  of  transients;  step, 
ramp,  decaying  exponential,  and  exponentially  decaying  sinusoid,  are  used.  The  magnitude 
of  the  transient  shift  Ay(CTy)  at  the  ‘250th  observation  is  varied  in  multiples  of  the  standard 
deviation  of  the  steady-state  data,  CTy  =  2.27.  The  transients  arc  shown  in  Figure  12  on 
page  116. 

For  both  truncation-point  selection  rules,  the  parameter  estimation  procedure  is  the 
same  and  the  MMAE  implementation  is  identical.  The  MMAE  filters  are  positioned  at  the 
mean  and  ±  3  [HP~H  +  R]^ .  The  estimated  system  parameters  are  shown  in  Table  55. 
The  scalar  fc  is  arbitrarily  set  to  one  in  this  Monte  Carlo  analysis. 

The  simple  rule  of  declaring  the  steady  state  when  the  mean  filter  had  an  MMAE 
probability  greater  than  ninety  percent,  Prob(/iy(t„))  >  0.9,  is  implemented.  The  truncation- 


Table  55.  Correlation  Technique  Parameter  Estimation  Summary 


Mv  y 

QSSHQuliMiEfil 

0.30  0.27  0.11 

0.57  0.80  0.05 
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Table  56.  AR(2)  Truncation  Point  Errors  {ho  —  Uo)  (Prob(/iy(t„))  >  0.9) 


Type 

Fail 

Min 

Max 

Avg 

St  Dev 

MSE 

AR(2)  Data  with  (j>i  =  0.6  and  <f)2 

=  0.3 

2.0 

Step 

-498 

177.8 

163.0 

58170.3 

2.0 

Ramp 

-212 

IBi 

147.2 

29589.1 

2.0 

Exp 

-184 

1146 

149.1 

136.1 

40754.2 

2.0 

Sine 

-498 

123 

161.9 

178696.9 

1.5 

Step 

0.00 

-498 

886 

75.8 

250.9 

68704.4 

1.5 

Ramp 

0.00 

-259 

1213 

67.9 

153.7 

28235.0 

1.5 

Exp 

0.00 

-289 

1093 

102.1 

168.5 

38802.7 

1.5 

Sine 

0.00 

-498 

226 

-410.4 

136.4 

186985.9 

1.0 

Step 

-498 

924 

-202.0 

296.5 

128721.8 

1.0 

Ramp 

-498 

868 

-18.7 

166.2 

27973.6 

1.0 

Exp 

-306 

995 

8.8 

161.3 

26083.1 

1.0 

Sine 

-498 

339 

-422.1 

118.5 

192186.5 

0.5 

Step 

0.00 

-498 

925 

-438.9 

123.6 

207922.1 

0.5 

Ramp 

0.00 

-498 

824 

-309.5 

195.4 

133990.1 

0.5 

Exp 

0.00 

-383 

1135 

-85.2 

154.6 

31145.5 

0.5 

Sine 

0.00 

-498 

3 

-433.6 

101.3 

198285.4 

Note:  Negative  Truncation  points  indicate  transient  observations  are  retained. 


point  errors  for  1,000  runs  are  summarized  in  Table  56.  In  this  table,  “Fail”  indicates  the 
rate  of  times  that  no  truncation  point  is  selected.  Since  the  “Fail”  column  is  all  zeros,  the 
algorithm  always  selected  a  truncation  point. 

The  desired  truncation-point  error  is  a  small  positive  number  so  that  all  of  the  transient 
data  is  truncated  and  little  excessive  truncation  of  steady-state  data  occurs.  Large  negative 
errors  indicate  that  many  transient  observations  are  retained  in  the  truncated  sequence  and 
they  may  bicis  estimates  of  steady-state  parameters.  Large  positive  truncation-point  errors 
indicate  that  many  steady-state  observations  are  truncated,  increasing  the  variability  of 
steady-state  parameter  estimates. 

The  truncation-point  algorithm  based  on  the  mean  filter’s  probability  generally  works 
well.  The  transients  with  larger  shifts,  with  the  exception  of  the  “Sine”,  results  in  small 
positive  truncation-point  errors.  The  transient  of  an  exponentially  decaying  sinusoid,  shown 
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Table  57.  AR(2)  Truncation  Point  hp  Evaluations  (Prob(/Zy(^n))  >  0.9) 


Transient 
A/i„(ay)  Type 

Fail 

Cov 

NOBM 

Half  Width 

Cov 

OBM 

Half  Width 

Bias  Test 
m)  xHctd 

i 

\R(2)  Data  with  <^i  = 

0.6  and  (f>2  =  0.3 

None 

0.02 

0.85 

0.392  ±  0.146 

0.84 

0.376  ±  0.124 

0.98 

0.93 

2,0 

Step 

0.05 

0.83 

0.414  ±  0.167 

0.82 

0.96 

0.93 

2.0 

Ramp 

0.03 

0.86 

0.85 

0.99 

0.95 

2.0 

Exp 

0.03 

0.82 

0.82 

0.97 

0.94 

2.0 

Sine 

0.03 

0.92 

0.448  ±  0.141 

0.63 

0.47 

1.5 

Step 

MM 

0.85 

0.412  ±  0.169 

0.82 

0.399  ±  0.150 

0.85 

0.82 

1.5 

Ramp 

0.83 

0.399  ±  0.159 

0.82 

0.386  ±  0.141 

0.98 

0.94 

1.5 

Exp 

0.84 

0.409  ±  0.167 

0.82 

0.394  ±  0.147 

0.96 

0.93 

1.5 

Sine 

0.445  ±  0.162 

0.88 

0.398  ±  0.090 

0.66 

0.48 

1.0 

Step 

Ml 

0.69 

0.439  ±  0.173 

0.67 

0.418  ±  0.152 

0.57 

0.43 

1.0 

Ramp 

EEi 

0.83 

0.392  ±  0.147 

0.81 

0.377  ±  0.130 

0.94 

0.91 

1.0 

Exp 

0.02 

0.85 

0.396  ±  0.150 

0.84 

0.382  ±  0.136 

0.98 

0.95 

1.0 

Sine 

0.02 

0.91 

0.441  ±  0.157 

0.87 

0.382  ±  0.087 

0.71 

0.54 

mm 

Step 

MM 

0.67 

0.66 

0.367  ±  0.119 

0.77 

0.55 

Ramp 

0.77 

0.75 

0.369  ±  0.110 

0.85 

0.70 

Exp 

0.84 

0.82 

0.371  ±  0.131 

0.97 

0.94 

Sine 

0.87 

0.83 

0.367  ±  0.080 

0.75 

0.58 

Note;  The  coverage  and  Schruben  estimation  accuracy  is  %  ±0.016  for  1000  runs. 


in  Figure  12,  repeatly  traverses  the  steady-state  mean.  When  the  transient  observations  pass 
near  the  estimated  steady-state  mean,  a  premature  truncation  point  is  sometimes  selected. 
The  transients  with  a  smaller  shift  in  the  mean,  A/Xj,(c7y),  are  more  difficult  to  detect. 

The  effect  of  the  truncation  point  on  the  ability  to  draw  inferences  can  be  seen  by 
constructing  confidence  intervals.  Transient  observations  in  truncated  sequences  often  bias 
the  mean  estimates,  resulting  in  lov-er  than  nominal  coverage  rate  for  the  confidence  intervals. 
In  contrast,  excess  truncation  reduces  the  precision  of  the  mean  estimate  and  increases  the 
confidence  interval  half  width. 

Table  57  shows  effectiveness  of  the  confidence  intervals  constructed  with  the  truncated 
sequences.  The  column  labeled  “Fail”  indicates  the  rate  of  times  that  the  truncation  se¬ 
quence  failed  to  obtain  statistically  independent  batches  with  Fishman’s  algorithm.  (See 
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Equation  (6)  on  page  27.) 

The  coverage  rates  and  average  confidence  interval  half  widths  are  excellent.  The  base¬ 
line  for  comparison  are  the  sequences  truncated  at  the  500th  observations.  These  sequences 
with  no  transient  data  are  labeled  under  transient  type  as  “None”.  These  single- run  con¬ 
fidence  interval  techniques  are  based  on  approximations  to  account  for  the  correlation  in 
the  output  sequence.  Although  few  of  the  truncated  sequences  result  in  nominal  confidence 
interval  coverage  rates  of  0.9,  they  are  as  close  as  the  baseline  case,  which  has  a  perfect  trun¬ 
cation  point.  Even  though  the  exponentially  decaying  sinusoid  transients  generally  result  in 
selected  truncation  points  prior  to  steady  state,  these  transients  contain  both  high  and  low 
data  values  which  apparently  cancel  each  other’s  effect.  The  transients  with  a  small  step 
change  to  steady  state  resulted  in  the  lowest  coverage  rates.  Perhaps  these  transient  values 
are  too  small  to  be  detected  but  significant  enough  to  bias  the  mean  estimate.  None  of  the 
cases  result  in  excessive  truncation,  so  the  average  confidence  interval  half  widths  are  also 
comparable  to  the  baseline  case. 

The  pass  rates  for  Schruben’s  initial  bias  test,  discussed  in  Hection  2.4.1,  are  also 
presented.  The  X^(^v))  Equation  (9),  rates  are  generally  near  the  nominal  rate  of  0.9. 
Schruben’s  tests  with  the  estimated  variance  F{al),  Equation  (10),  provide  questionable 
indications  since  the  mean  estimate  variance  d?  is  based  on  the  biased  estimates  of  Qd  and 
R  (See  page  167). 

Table  58  shows  the  average  number  of  batches  and  batch  size  selected  by  Fishman’s 
algorithm,  Equation  (6).  Little  variation  between  the  different  transient  types  is  apparent. 

In  contrast  to  monitoring  the  MMAE  filter  probabilities  to  determine  a  truncation 
point,  the  MMAE  time-varying  mean  estimate  Av(^n)  can  be  tracked.  For  a  test  of  this 
approach,  the  truncation  point  is  selected  when  the  MMAE  mean  estimate  /iy(tn)  is  within 
1.0  of  the  steady-state  mean  estimate  y  for  5  consecutive  observations.  The  results  of  this 
strategy  are  shown  in  Taole  56. 
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Table  58. 


Fishman’s  Number  and  Size  of  Batches  (Prob(/ij,(t„))  >  0.9) 


Transient 

Number 

of  Batches 

Batch 

Sizes 

_ 1 

Mat 

Ab 

None 

32.83 

13.32 

56.66 

31.47 

2.0 

Step 

12.06 

51.25 

26.50 

2.0 

Ramp 

31.52 

13.16 

54.72 

29.27 

2.0 

Exp 

31.42 

12.24 

52.20 

28.15 

2.0 

Sine 

42.57 

19.25 

59.34 

37.47 

1.5 

Step 

31.13 

12.18 

51.75 

27.65 

1.5 

Ramp 

13.23 

53.53 

28.86 

1.5 

Exp 

31.38 

12.64 

53.72 

27.78 

1.5 

Sine 

38.38 

17.59 

63.55 

34.88 

1.0 

Step 

28.77 

67.21 

35.71 

1.0 

Ramp 

32.84 

54.96 

27.47 

1.0 

Exp 

32.35 

56.01 

28.51 

1.0 

Sine 

36.99 

15.47 

62.09 

28.91 

0.5 

Step 

34.74 

16.35 

69.90 

35.17 

0.5 

Ramp 

33.49 

14.69 

65.55 

32.86 

0.5 

Exp 

34.25 

14.23 

56.06 

28.08 

0.5 

Sine 

35.35 

12.82 

61.22 

21.96 
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Table  59.  AR(2)  Truncation  Point  Errors  (no  —  rio)  (ABS(/iy(t„)  —  y)  <  1) 


Type 

Fail 

Min 

Max 

Avg 

St  Dev 

MSE 

AR(2)  Data  with  4>i  =  0- 

5  and  ^2 

=  0.3 

2.0 

Step 

0.00 

-499 

1232 

108.0 

147.4 

33385.6 

2.0 

Ramp 

0.00 

-214 

734 

35.4 

128.4 

17741.0 

2.0 

Exp 

0.00 

-206 

887 

92.9 

115.7 

22025.9 

2.0 

Sine 

0.00 

-499 

103 

-438.4 

108.2 

203952.7 

1.5 

Step 

0.00 

-499 

788 

1.5 

244.2 

59654.7 

1.5 

Ramp 

0.00 

-323 

1223 

14.2 

142.0 

20359.6 

1.5 

Exp 

0.00 

-320 

884 

45.9 

153.4 

25628.1 

1.5 

Sine 

0.00 

-499 

114 

-451.7 

89.1 

211956.7 

1.0 

Step 

0.00 

-499 

828 

-279.3 

265.0 

148225.7 

1.0 

Ramp 

0.00 

-499 

733 

-73.8 

153.3 

28933.8 

1.0 

Exp 

0.00 

-330 

768 

-43.6 

144.4 

22741.9 

1.0 

Sine 

0.00 

-499 

230 

-456.4 

79.5 

214582.1 

0.5 

Step 

-499 

650 

-462.9 

93.4 

223054.0 

0.5 

Ramp 

Bl 

-499 

620 

-357.0 

170.0 

156352.2 

0.5 

Exp 

0.00 

-393 

914 

-137.8 

138.2 

38100.2 

0.5 

Sine 

0.00 

-499 

-81 

-463.3 

64.8 

218834.9 

Note:  Negative  Truncation  points  indicate  transient  observations  are  retained. 
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By  comparing  Tables  56  and  59,  the  truncation-point  errors  for  both  MMAE  ap¬ 
proaches  are  seen  to  be  very  similar.  The  similarity  is  not  surprising  since  the  MMAE 
probability  approach  tracked  a  probability  while  the  MMAE  mean  approach  tracked  the  filter 
means  weighted  by  their  probabilities.  Therefore,  both  approaches  are  based  on  the  MMAE 
filter  probabilities.  The  truncation-point  selection  approach  based  on  ABS(/ij,(t„)  —  y)  <  1.0 
appears  to  work  slightly  better  with  the  transients  with  larger  shifts,  such  as  A/i„((7„)  of  1.5 
or  2.0.  In  contrast,  the  approach  based  on  Prob(/iy(i„)  =  y)  >  0.9  has  smaller  truncation- 
point  errors  for  the  transients  with  a  relatively  smaller  shift  in  their  mean.  However,  any 
tradeoffs  between  the  two  approaches  is  very  dependent  upon  the  arbitrary  selected  criteria 
of  0.9  and  1.0. 

As  seen  from  comparing  Table  57  and  Table  60,  the  confidence  interval  coverage  rates 
and  half  widths  are  very  similar  between  the  two  truncation-point  selection  rules.  Since  there 
is  minimal  change  in  the  number  or  size  of  batches,  they  are  not  shown.  Because  the  two 
truncation-point  selection  approaches  have  such  similar  results,  only  one  is  used  for  further 
testing.  The  rule  based  on  two  mean  estimates  being  within  a  small  tolerance  has  more 
potential  for  determining  a  heuristic  to  select  the  tolerance  than  the  rule  based  on  a  filter 
probability  exceeding  an  arbitrary  level.  Therefore,  the  next  section  applys  the  truncation- 
point  selection  rule  based  on  ABS{^y{tn)  —  y)  <  1  to  actual  discrete-event  simulation  output. 

5.4-2  M/M/l  Queue  Waiting  Time  Data.  Detecting  the  transient  in  the  time-series 
test  cases  is  considerably  easier  than  with  a  single  simulation  output  sequence.  In  some 
actual  simulation  outputs,  rather  than  an  initial  bias  consisting  of  observations  that  are 
statistical  outliers  of  the  steady-state  output  distributions,  the  initial  transient  consists  of 
observations  from  a  similar,  but  skewed  distribution,  which  varies  with  time.  Transient 
values  can  not  be  identified  simply  by  being  statistical  outliers  of  an  estimated  steady-state 
output  distribution.  But  the  mean  of  a  group  of  transient  values  may  be  biased  compared 
to  the  mean  of  a  group  of  steady-state  observations. 

In  Section  2.4.3,  Figures  2  through  4  depict  the  empirical  transient  for  the  simulation  of 
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Table  60.  AR(2)  Truncation  Point  Ho  Evaluations  (ABS(/iy  —  y)  <  1) 


Transient 
Afiy{ay)  Type 

Cov 

OBM 

Half  Width 

Bias  Test 

AR(2)  Data  with  <f>i  = 

0.6  and  (f)2  =  0.3 

None 

0.02 

0.85 

0.392  ±  0.146 

0.84 

0.376  ±  0.124 

0.98 

0.93 

2.0 

Step 

IjtjlJ 

0.84 

0.407  ±  0.162 

0.83 

0.394  ±  0.145 

0.95 

0.92 

2.0 

Ramp 

0.87 

0.398  ±  0.154 

0.86 

0.382  ±  0.134 

0.99 

0.95 

2.0 

Exp 

0.03 

0.83 

0.404  ±  0.152 

0.82 

0.389  ±  0.131 

0.97 

0.95 

2.0 

Sine 

0.02 

0.92 

0.455  ±  0.136 

0.92 

0.428  ±  0.092 

0.61 

0.44 

1.5 

Step 

My 

0.84 

0.410  ±  0.169 

0.82 

0.395  ±  0.149 

IBI 

0.78 

1.5 

Ramp 

EES 

0.83 

0.398  ±  0.154 

0.81 

0.382  ±  0.134 

ly 

0.94 

1.5 

Exp 

0.02 

0.84 

0.405  ±  0.158 

0.83 

0.390  ±  0.140 

m 

0.93 

1.5 

Sine 

0.02 

0.91 

0.450  ±  0.157 

0.88 

0.402  ±  0.085 

0.63 

0.43 

1.0 

Step 

0.65 

0.444  ±0.173 

0.62 

0.422  ±  0.151 

0.49 

0.34 

1.0 

Ramp 

0.82 

0.387  ±  0.144 

0.80 

0.372  ±  0.124 

0.92 

0.89 

1.0 

Exp 

EH 

0.85 

0.392  ±  0.153 

0.84 

0.378  ±  0.137 

0.97 

0.94 

1.0 

Sine 

0.02 

0.92 

0.446  ±  0.156 

0.87 

0.384  ±  0.084 

0.69 

0.51 

0.5 

Step 

nay 

0.67 

0.383  ±  0.130 

0.65 

0.366  ±  0.118 

0.76 

0.52 

0.5 

Ramp 

0.77 

0.388  ±  0.128 

0.74 

0.370  ±  0.111 

0.84 

0.65 

0.5 

Exp 

0.03 

0.83 

0.381  ±  0.142 

0.81 

0.366  ±  0.125 

0.96 

0.93 

0.5 

Sine 

0.01 

0.89 

0.430  ±  0.128 

0.84 

0.367  ±  0.075 

0.73 

0.56 

Note:  The  coverage  and  Schruben  estimation  accuracy  is  %  ±0.016  for  1000  runs. 
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Table  61.  Corre 

ation  Techniq 

ue  Parameter  Estimation  Summary 

y 

Ay  ^y 

<^1 

Ai, 

<f>2 

k 

Ajfc  H 

p  =  0.95 

0.95  0.04 

0.04  0.04 

1.00  0.00 

an  M/M/1  queue  with  traffic  intensity  p  of  0.95.  The  transient  can  not  be  detected  visually 
from  a  single  run.  After  applying  Welch’s  moving  window  technique  to  the  average  of  30 
simulation  replications,  shown  in  Figure  4,  the  average  end  of  transient  is  seen  to  be  about 
1,000  observations. 

The  same  truncation  selection  procedure  as  applied  on  the  time-series  data  is  applied 
to  simulation  output  of  M/M/1  with  a  traffic  intensity  p  of  0.95.  As  Howard  [37]  reports, 
the  same  MMAE  filter  spacing  does  not  work  well  with  different  models.  The  MMAE  filter 
spacing  is  significantly  increased  for  the  M/M/1  output  sequences.  The  filters  are  positioned 
based  on  the  minimum,  mean,  and  maximum  of  the  last  half  of  the  output  sequence: 


=  y  +  0.9(y  -  8j)  where  6,  =  y,  and  ymax  (67) 

The  simulation  model  inputs  are  such  that  the  traffic  intensity  p  is  0.95  and  the  the¬ 
oretical  steady-state  waiting  time  is  10.0  time  units.  The  parameter  estimates  for  2,000 
simulation  runs  are  shown  in  Table  61.  Generally,  the  output  sequence  is  represented  ap¬ 
proximately  as  a  pure  AR(1).  The  system  model  is  nearly  a  pure  AR(1)  since  ^2  is  near 
zero  and  the  estimated  measurement  noise  variance  R  is  zero  {k  =  1.0).  Originally,  it  was 
hoped  that  the  measurement  noise  might  represent  the  “lack  of  fit”  for  assuming  an  AR(2) 
dynamics  equation.  However,  this  example  seems  to  indicate  that,  if  the  dynamics  model 
inadequately  represents  the  data’s  correlation  structure,  the  parameter  estimation  increases 
the  Kalman  filter  gain  to  one  in  order  to  disregard  the  dynamics  model  predictions  com¬ 
pletely.  But  a  Kalman  filter  gain  of  one  indicates  no  measurement  noise  is  needed  to  model 
the  data.  Thus,  the  more  “lack  of  fit”  of  the  dynamics  model  to  the  real  data  there  is,  the 
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less  measurement  noise  variance  is  estimated. 

The  average  selected  truncation  point,  shown  in  Table  62,  is  1,321.6.  This  compaires 
reasonably  well  with  a  truncation  point  of  1,000  identified  by  Welch’s  technique. 

As  shown  in  Table  63,  the  Non- Overlapping  Batch  Means  (NOBM)  and  Overlapping 
Batch  Means  (OBM)  coverage  rates  with  the  truncated  sequences  are  only  0.56  and  0.55 
for  90  percent  nominal  confidence  intervals.  Two  possible  reasons  for  the  low  coverage  are 
biased  mean  estimates  or  Fishman’s  algorithm,  which  selects  batch  size,  induces  a  bias.  A 
biased  mean  estimate  is  unlikely  since  10,000  observations  are  used  and  the  average  trun¬ 
cation  point  of  1321.6  is  greater  than  the  truncation  point  selected  by  Welch’s  algorithm. 
Fishman’s  algorithm  appears  to  have  difficulty  with  this  output  data  since  for  32  percent 
of  the  sequences  no  batch  size  is  determined.  These  confidence  intervals  are  constructed 
with  number  of  batches  summarized  in  Table  64.  The  effects  of  Fishman’s  algorithm  are 
investigated  further. 


Table  64. 

?'ishman’s  Number  aii 

d  Size  of  Batches 

Number  of  Batches 
Am  d’M 

Batch  Sizes 

Ab  ctb 

p  =  0.9 

17.46  7.89 

601.44  256.68 
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Table  65  shows  the  empirical  mean  of  the  output  distribution  for  every  250th  entity 
through  the  simulated  queue.  The  mean  of  the  1,000th  observation  is  very  near  the  theo¬ 
retical  steady-state  mean  of  10.0,  and  the  transient  appears  to  be  effectively  complete  by 
the  2,000th  observation.  Therefore,  the  average  truncation  point  of  1321.6  retains  some 
biased  data,  but  the  magnitude  of  the  bias  appears  to  have  been  small.  From  examining 
the  variation  in  the  data  over  the  2,000  runs,  the  average  selected  truncation  point  seems 
reasonable. 

Since  the  analytical  alternatives  for  truncation-point  selection  are  unreliable  heuristics 
[24,  105],  one  evaluation  approach  is  to  compare  the  Kalman  filter  results  to  the  “best 
possible” .  The  “best  possible”  is  determined  by  increasing  the  truncation  point  at  increments 
and  calculating  the  confidence  intervals  on  systems  with  analytic  steady-state  values. 

Periodic  truncation  points  at  every  250th  observation  are  tested,  and  the  results  are 
shown  in  Table  66  for  2,000  replications.  The  first  two  columns  are  the  mean  estimates 
and  associated  standard  deviation  for  each  of  the  truncated  sequences.  The  column  labeled 
is  the  mean  of  the  observations  in  the  truncated  sequence,  and  is  the  standard 
deviation  over  the  2,000  replications.  These  columns  indicate  the  bias  for  not  truncating  any 
observations  is  very  small,  and  that  small  bias  is  removed  with  truncating  several  hundred 
initial  data  values.  The  coverage  rates,  for  any  of  these  truncation  sequences,  are  significantly 
below  the  nominal  rate  of  90  percent.  This  may  be  due  to  the  highly  correlated  nature  of 
the  output  sequences,  the  run  length,  or  Fishman’s  algorithm. 

Table  66  indicates  that  Fishman’s  algorithm.  Equation  (6),  to  determine  the  number  of 
statistically  independent  batches  may  induce  some  bias.  At  each  of  the  horizontal  lines,  the 
rate  of  sequences  failing  Fishman’s  test  for  independence  significantly  increases.  A  probable 
cause  of  these  increases  is  that  Fishman’s  algorithm  only  considers  batch  sizes  in  powers 
of  two.  Therefore,  as  more  data  is  truncated,  the  truncated  data  sequence  is  shorter,  and 
some  of  the  sequences  have  insufficient  data  to  test  for  batch  independence  with  double  the 
batch  size.  At  the  same  place  that  the  failure  rate  increases,  the  coverage  rate  significantly 
decreases.  Possibly,  truncated  sequences  with  large  and  highly  correlated  values  fail  the  test 


192 


Table  65.  Average  Periodic  Output  Values  for  2,000  Replications  (/i„  =  10. 


n 

Avn 

1 

0.00 

0.00 

250 

6.17 

5.80 

500 

7.67 

7.39 

750 

8.60 

8.46 

1000 

9.25 

8.98 

1250 

9.73 

9.66 

1500 

9.61 

9.70 

1750 

9.73 

10.00 

2000 

9.90 

10.27 

2250 

10.08 

10.43 

2500 

9.83 

10.28 

2750 

10.00 

10.30 

3000 

10.02 

10.40 

3250 

10.11 

10.55 

3500 

9.54 

10.14 

3750 

10.00 

10.44 

4000 

10.24 

10.54 

4250 

9.93 

10.16 

4500 

9.66 

9.95 

4750 

9.61 

10.09 

5000 

9.36 

10.15 

5250 

9.60 

10.42 

5500 

10.08 

10.87 

5750 

10.04 

10.70 

6000 

10.11 

10.55 

6250 

9.98 

10.71 

6500 

9.74 

10.70 

6750 

9.91 

10.88 

7000 

9.95 

10.96 

7250 

10.20 

11.20 

7500 

10.17 

11.17 

7750 

10.38 

11.00 

8000 

10.33 

11.15 

8250 

10.41 

11.22 

8500 

10.36 

11.42 

8750 

10.50 

10.93 

9000 

10.14 

10.64 

9250 

9.85 

10.33 

9500 

9.77 

10.09 

9.60 

9.83 
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Table  66.  Periodic  Truncation  Points  (Fishman’s  Number  of  Batches) 


■■ 

mm 

Fishman’s  Algorithm 

NOBM 

OLMB 

Bia.s 

no 

Fails 

Batches 

Size 

Cov 

HW 

Cov 

HW 

Test 

9.66 

3.80 

Q3 

16.27 

756.55 

0.62 

3.172 

0.61 

3.069 

0.42 

9.80 

3.89 

16.13 

748.54 

0.62 

3.237 

0.61 

3.128 

0.45 

9.88 

3.98 

15.88 

725.23 

0.63 

3.279 

0.61 

3.138 

0.48 

9.92 

15.78 

709.65 

0.64 

3.265 

0.62 

3.124 

0.49 

1000 

9.95 

4.14 

0.47 

19.84 

486.42 

0.51 

2.221 

0.49 

2.159 

0.50 

1250 

9.96 

4.22 

0.48 

20.27 

465.38 

0.47 

2.163 

0.46 

2.115 

0.51 

1500 

9.97 

0.45 

19.21 

479.60 

0.49 

2.267 

0.47 

2.194 

0.52 

1750 

9.99 

4.38 

0.46 

19.40 

460.95 

0,48 

2.242 

0.47 

2.188 

0.52 

2000 

9.99 

4.45 

0.42 

18.43 

475.01 

0.50 

2.379 

0.49 

2.282 

0.53 

2250 

9.99 

4.53 

0.44 

18.65 

454.59 

0.47 

2.340 

0.46 

2.259 

0.54 

2500 

9.99 

4.61 

0.40 

17.74 

467.98 

0.51 

2.465 

0.49 

2.367 

0.55 

2750 

4.69 

0.41 

17.92 

447.73 

0.48 

2.438 

0.47 

2.346 

0.56 

|({jmi| 

9.99 

4.77 

0.37 

16.97 

462.66 

0.52 

2.563 

0.51 

2.446 

0.57 

9.99 

4.86 

0.38 

17.07 

442.38 

0.50 

2.540 

0.48 

2.436 

0.57 

9.99 

4.95 

0.34 

16.18 

454.98 

0.53 

2.685 

0.51 

2.557 

0.58 

0.36 

16.21 

434.75 

0.52 

2.670 

0.50 

2.547 

0.40 

9.99 

5.16 

0.32 

15.21 

452.37 

0.54 

2.831 

0.52 

2.675 

0.38 

4250 

9.99 

5.27 

0.32 

15.22 

429.58 

0.53 

2.804 

0.51 

2.655 

0.42 

EMIIJ 

9.99 

5.38 

0.28 

14.37 

444.77 

0.54 

2.974 

0.52 

2.807 

0.46 

■MM 

5.49 

0.28 

14.24 

423.19 

0.52 

2.963 

0.50 

2.820 

0.51 

laiiiiii 

5.61 

0.25 

13.55 

435.26 

0.53 

3.168 

0.51 

3.016 

0.56 

5250 

IB 

5.72 

0.25 

13.60 

408.13 

0.53 

3.154 

0.50 

2.977 

0.60 

5500 

10.06 

5.84 

0.66 

18.59 

252.00 

0.24 

1.823 

0.23 

1.769 

0.65 

5750 

10.06 

5.94 

0.63 

17.92 

249.53 

0.27 

1.888 

0.25 

1.831 

0.67 

6000 

10.06 

6.06 

0.60 

16.88 

249.92 

0.28 

1.987 

0.26 

1.909 

0.68 

6250 

10.07 

6.17 

0.57 

16.20 

246.80 

0.29 

2.074 

0.28 

1.993 

0.70 

6500 

10.07 

6.29 

0.55 

15.24 

246.86 

0.31 

2.168 

0.30 

2.075 

0.73 

6750 

10.09 

6.41 

0.52 

14.52 

243.69 

0.33 

2.271 

0.32 

2.168 

0.75 

7000 

10.10 

6.55 

0.49 

13.72 

241.42 

0.35 

2.392 

0.33 

2.276 

0.77 

7250 

10.12 

6.68 

0.44 

12.77 

240.18 

0.38 

2.574 

0.36 

2.434 

0.80 

7500 

10.13 

6.84 

0.40 

11.77 

238.75 

0.40 

2.768 

0.38 

2.614 

0.83 

7750 

10.11 

7.00 

0.80 

17.82 

128.89 

0.08 

1.453 

0.08 

1.414 

0.86 

8000 

10.08 

7.16 

0.75 

16.61 

126.06 

0.12 

1.566 

0.11 

1.506 

0.88 

8250 

10.04 

7.31 

0.69 

14.66 

125.69 

0.13 

1.702 

0.12 

1.627 

0.91 

8500 

9.98 

7.42 

0.61 

12.86 

125.01 

0.17 

1.927 

0.16 

1.831 

0.95 

8750 

9.88 

7.61 

0.52 

1 _ _ _ 

11.07 

123.50 

0.22 

2.198 

0.21 

2.076 

0.98 

9000 

9.76 

7.85 

0.84 

15,83 

64.24 

0.02 

1.212 

0.02 

1.161 

0.09 

9250 

9.67 

3.18 

0.70 

12.36 

63.92 

0.05 

1.471 

0.05 

1.395 

0.93 

8.56 

15.46 

32.51 

0.01 

0.869 

0.01 

0.838 

0.98 
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Table  67.  Truncation  Points  Up  (10  Batches) 


Type 

Fail 

Min  Max 

Avg  St  Dev 

0.95 

0.001 

7  8070 

1364.7  1260.4 

Table  68.  Truncation  Point  no  Evaluations  (10  Batches) 


■ 

Cov 

OBM 

Half  Width 

Bias  Test 

m) 

0.95 

0.00 

0.64  3.785  ±  3.870 

0.63 

3.685  ±  3.783 

0.54 

for  independent  batches  more  often.  As  seen  in  Figure  2  on  page  38,  simulation  output  of 
M/M/1  queue  with  p  =  0.95  occasionally  has  very  large  and  highly  correlated  subsequences. 
In  contrast,  sequences  with  generally  low  values,  often  with  idle  servers  causing  independent 
subsequences,  may  tend  to  pass  Fishman’s  test  for  independent  batches.  Since  low  biased 
sequences  may  pass  Fishman’s  test  and  be  used  to  form  confidence  intervals  at  a  higher 
frequency,  the  coverage  rates  are  significantly  less  than  the  nominal  rate. 

To  eliminate  the  effects  of  Fishman’s  algorithm  determining  the  number  of  statisti¬ 
cally  independent  batches,  one  thousand  replications  are  tested  with  the  number  of  batches 
fixed  to  ten.  The  truncation-point  algorithm  is  applied  to  these  1,000  simulation  runs. 
The  selected  truncation  points,  summarized  in  Table  67,  are  similar  to  the  previous  test. 
However,  Table  68  shows  that  fixing  the  number  of  batches  at  ten  improved  the  coverage 
rate.  Therefore,  Fishman’s  algorithm  to  select  batch  sizes  does  induce  a  bias  and  reduces 
the  coverage  rates. 

The  periodic  results  with  a  fixed  batch  sii,e  of  ten,  shown  in  Table  69,  have  a  higher 
coverage  rate  at  every  periodic  truncation  point.  However,  no  significant  degradation  in 
coverage  rate  is  apparent  when  the  initial  transient  is  not  eliminated.  The  M/M/1  with 
p  =  0.95  queuing  model  was  selected  because  of  its  extreme  positive  correlation.  The  high 
correlation  induces  a  relatively  longer  transient  than  other  simulation  models.  Perhaps  the 
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Table  69.  Periodic  Truncation  Points  (Number  of  Batches  Fixed  at  10) 


no 

Fails 

Batches 

Size 

NORM 

OLMB 

Schrub 

0 

9.70 

3.87 

0.00 

1000.00 

r  0.65 

3.654 

0.64 

3.552 

0.41 

250 

9.83 

3.97 

0.00 

975.00 

0.65 

3.713 

0.65 

3.588 

0.45 

500 

9.91 

4.06 

0.00 

950.00 

0.66 

3.756 

0.65 

3.619 

0.48 

750 

9.96 

4.14 

0.00 

925.00 

0.66 

3.787 

0.65 

3.643 

0.49 

1000 

9.99 

4.23 

0.00 

10.00 

900.00 

0.65 

3.804 

0.64 

3.664 

0.50 

1250 

10.02 

4.31 

0.00 

10.00 

875.00 

0.66 

3.823 

0.64 

3.681 

0.50 

1500 

10.03 

4.39 

0.00 

10.00 

850.00 

0.65 

3.859 

0.64 

3.695 

0.51 

1750 

10.05 

4.47 

0.00 

10.00 

825.00 

0.65 

3.895 

0.63 

3.703 

0.51 

2000 

10.06 

4.55 

0.00 

10.00 

800.00 

0.64 

3.890 

0.62 

3.707 

0.53 

2250 

10.07 

4.63 

0.00 

10.00 

775.00 

0.64 

3.879 

0.63 

3.710 

0.54 

2500 

10.07 

4.73 

0.00 

10.00 

750.00 

0.64 

3.882 

0.62 

3.714 

0.55 

2750 

10.08 

4.82 

0.00 

10.00 

725.00 

0.64 

3.908 

0.62 

3.718 

0.55 

3000 

10.08 

4.91 

0.00 

10.00 

700.00 

0.64 

3.911 

0.62 

3.725 

0.55 

3250 

10.08 

5.01 

0.00 

10.00 

675.00 

0.63 

3.903 

0.62 

3.734 

0.57 

3500 

10.10 

5.11 

0.00 

10.00 

650.00 

0.63 

3.916 

0.61 

3.740 

0.58 

3750 

10.11 

5.21 

0.00 

10.00 

625.00 

0.63 

3.940 

0.61 

3.740 

0.39 

4000 

10.11 

5.31 

0.00 

10.00 

600.00 

0.62 

3.932 

0.60 

3.740 

0.39 

4250 

10.11 

5.42 

0.00 

10.00 

575.00 

0.60 

3.914 

0.58 

3.747 

0.42 

4500 

10.11 

5.54 

0.00 

10.00 

550.00 

0.59 

3.934 

0.57 

3.753 

0.45 

4750 

10.14 

5.68 

0.00 

10.00 

525.00 

0.58 

3.941 

0.55 

3.756 

0.50 

10.18 

5.82 

500.00 

0.56 

3.934 

0.54 

3.757 

0.56 

5250 

10.19 

5.97 

475.00 

0.56 

3.946 

0.53 

3.751 

0.60 

5500 

10.20 

6.12 

450.00 

0.55 

3.939 

0.52 

3.746 

0.65 

5750 

10.21 

6.26 

BmniiB 

425.00 

0.56 

3.941 

0.54 

3.749 

0.67 

10.22 

10.00 

400.00 

0.55 

3.938 

0.54 

3.746 

0.69 

6250 

10.22 

6.54 

10.00 

375.00 

0.55 

3.937 

0.53 

3.730 

0.72 

6.67 

10.00 

350.00 

0.54 

3.904 

0.52 

3.706 

0.74 

6750 

10.19 

10.00 

325.00 

0.53 

3.900 

0.51 

3.687 

0.75 

10.18 

6.94 

10.00 

300.00 

0.51 

3.871 

0.49 

3.664 

0.77 

7250 

10.00 

275.00 

0.51 

3.848 

0.49 

3.637 

0.80 

7500 

10.22 

7.18 

0.00 

H 

0.50 

3.820 

0.48 

3.594 

0.82 

7750 

10.20 

7.27 

0.00 

B 

0.46 

3.765 

0.44 

3.537 

0.85 

8000 

10.16 

7.33 

0.00 

B  BIB 

mmam 

0.44 

3.693 

0.42 

3.467 

0.88 

8250 

10.12 

7.38 

0.00 

10.00 

175.00 

0.44 

3.610 

0.42 

3.380 

0.91 

8500 

10.08 

7.44 

0.00 

10.00 

150.00 

0.43 

3.504 

0.41 

3.278 

0.95 

8750 

9.99 

7.55 

0.00 

10.00 

125.00 

0.39 

3.367 

0.37 

3.149 

0.99 

9000 

9.88 

7.75 

0.00 

10.00 

100.00 

0.36 

3.167 

0.34 

2.940 

0.09 

9250 

9.77 

8.08 

0.00 

10.00 

75.00 

0.30 

2.879 

0.29 

2.676 

0.93 

9500 

9.76 

8.48 

10.00 

50.00 

0.24 

2.540 

0.23 

2.357 

0.98 

9750 

9.70 

8.99 

10.00 

25.00 

0.16 

1.999 

0.15 

1.849 

1.00 
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initial  transient  is  insignificant  when  the  single-run  confidence  interval  techniques  are  applied 
to  one  long  simulation  r>in. 

5.5  Summary 

This  chapter  develops  and  tests  a  truncation-point  selection  algorithm  for  a  single 
simulation  output  sequence.  An  AR(2)  with  mccisurement  noise  model  is  estimated  using 
the  correlation  technique.  Based  on  the  estimated  model,  MMAE  is  applied  with  various 
mean  estimates.  The  truncation  point  is  selected  by  either  monitoring  the  probability  of  the 
filter  with  the  mean  estimate  or  by  tracking  the  distance  between  the  MMAE  mean  estimate 
and  the  assumed  steady-state  mean  estimate.  The  method  performs  well  on  time-series  data 
with  induced  transients  and  on  output  of  simulations  of  an  M/M/1  queue. 

For  simulation  practitioners,  passing  Fishman’s  test  for  statisically  independent  batches 
may  not  be  sufficient  to  ensure  reliable  estimates.  In  fact,  as  shown  in  the  example  for 
simulations  of  an  M/M/1  queue,  the  sequences  which  passed  Fishman’s  test  often  are  biased 
low.  As  Schmeiser  [83]  states,  “the  run  must  be  long  enough  to  calculate  a  valid  confidence 
interval.”  Passing  Fishman’s  test  does  not  ensure  that  the  run  length  is  long  enough. 

The  effective  attainment  of  steady  state  is  determined  based  on  a  single  simulation 
sequence.  This  seems  to  contradict  Law  and  Kelton’s  [53:550]  supposition  that  generally 
one  run  is  not  sufficient  to  estimate  the  truncation  point.  However,  the  transient’s  effect  on 
drawing  inferences  may  not  be  significant  when  confidence  interval  techniques  use  one  long 
simulation  run.  The  output  of  a  simulation  of  an  M/M/1  queue  with  traffic  intensity  p  of 
0.95  was  selected  because  the  high  correlation  should  induce  a  relative  long  transient.  But 
the  initial  transient  is  not  found  significant  in  decreasing  the  coverage  rate  when  a  single  long 
run  is  used.  Schmeiser  [83]  states  that  “initial  transients  are  often  a  major  factor  when  using 
independent  replications.”  Therefore,  an  approach  for  multiple  replications,  very  similar  to 
this  technique,  is  proposed  and  tested  in  Chapter  VI. 
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VI.  Multiple  Replications  Steady-State  Identification:  Multiple  Model 
Adaptive  Estimation  (MMAE)  Approach 


6. 1  Introduction 

Besides  the  confidence  interval  approaches  based  on  one  long  output  sequence,  confi¬ 
dence  intervals  can  be  constructed  using  replications  of  the  simulation  model  with  different 
random  number  generator  seeds.  The  confidence  interval  technique  is  called  the  method  of 
replications.  (See  page  26  for  a  discussion.)  The  advantage  of  the  method  of  replications 
is  that  means  of  the  replications  are  independent,  so  classical  statistic  techniques  can  be 
applied.  The  disadvantage  is  that  each  run  passes  through  the  transient  phase.  Because  of 
the  increase  in  transient  data,  selection  of  an  appropriate  truncation  point  becomes  more 
critical  to  prevent  bias  in  the  overall  mean  estimate. 

This  chapter  describes  the  MMAE  algorithm  to  select  the  truncation  points  for  multiple 
replications  of  univariate  sequences.  Evaluation  measures  are  discussed,  and  a  Monte  Carlo 
analysis  with  twelve  simulation  models  is  presented. 

6.S  Methodology 

The  methodology  applied  in  this  approach  is  the  same  as  the  MMAE  technique  for  a 
single  run  developed  in  Chapter  V.  The  only  change  is  that  the  single  sequence  is  replaced 
with  the  sequence  of  averages  from  the  M  independent  replications.  The  parameter  estima¬ 
tion,  discussed  in  the  Section  5.2,  is  applied  again.  Since  a  steady-state  model  is  needed,  the 
last  half  of  the  averaged  output  sequence  is  assumed  to  have  effectively  attained  steady  state 
and  is  used  to  estimate  the  steady-state  parameters.  Schruben  [87]  also  uses  the  last  half  of 
the  output  sequence  for  estimating  a  steady-state  model.  The  MMAE  algorithm  developed 
in  Section  5.3  is  applied  again.  After  the  estimated  truncation  point  ho  is  selected,  each  of 
the  M  sequences  is  truncated.  The  steps  of  the  truncation  point  selection  algorithm  follow: 
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Step  1.  Select  the  simulation  output  length  N  and  number  of  replications  M.  Run  M 
replications  of  the  simulation  model  until  N  outputs  occur  and  store  the  sequences  of 
outputs. 

Step  2.  Construct  a  sequence  of  length  N  of  the  averages  of  the  M  replications. 

Step  3.  Based  on  the  last  half  of  the  averaged  sequence,  determine  an  estimate  of  the 
steady-state  mean  y  with  Equation  (63)  and  the  autocorrelation  coefficients  with  Equa¬ 
tion  (64).  Based  on  Equations  (62)  and  (65),  estimate  ^2,  and  k.  (Initial  Monte 
Carlo  results  used  A:  =  1.0  as  recommended  in  Chapter  V,  but  better  results  are  ob¬ 
tained  by  using  k.) 

Step  4.  Apply  the  estimated  Kalman  filter  to  the  last  half  of  the  average  sequence  and  esti¬ 
mate  the  residual  variance  with  the  mean  squared  residuals,  as  shown  in  Equation  (66). 
Find  the  minimum  and  majcimum  observation  in  the  last  half  of  the  sequence  of  aver¬ 
ages.  Initialize  an  MMAE  bank  of  three  filters  spaced  according  to  minimum,  mean  y, 
and  maximum,  as  shown  in  Equation  (67).  Calculate  the  Kalman  filter  gain  K  with 
Equation  (51).  The  initial  state  estimates  x(to)  are  zero  vectors,  and  the  initial  MMAE 
filter  probabilities  Pj(io)  are  j. 

Step  5.  Process  the  observations  in  the  averaged  sequence  through  the  MMAE  bank  of 
filters.  For  each  observation,  propagate  the  estimated  state  vector  in  each  filter  using 
Equations  (22)  and  (45).  Update  the  state  estimates  with  Equation  (25)  using  the 
observation  minus  that  filter’s  mean  as  the  measurement.  Calculate  the  MMAE  filter 
probabilities  Pj(in)  with  Equation  (36).  Using  the  MMAE  filter  probabilities,  calculate 
the  weighted  MMAE  mean  estimate  Py(tn)  by  Equation  (39)  where  a(tn)  =  Av(^n)- 
When  ABS[Ay(^n)  v]  is  less  than  a  small  tolerance  for  a  few  consecutive  observations, 
select  the  truncation  point  ho  as  t„.  (In  these  tests,  the  small  tolerance  is  1.0  and  5 
consecutive  observations  are  used.  A  relative  criterion,  such  as  the  absolute  difference 
being  less  than  a  small  percentage  of  the  sample  mean,  is  probably  more  appropriate 
in  general  applications.)  Before  iterating,  reset  any  near-zero  filter  probabilities  to  a 
very  small  tolerance  to  allow  for  continue  adaptation  in  the  MMAE  algorithm. 
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The  selected  truncation  point  tiq  can  be  applied  to  the  M  output  sequences  used  in 
the  algorithm  or  the  truncation  point  can  be  applied  to  future  replications.  Pilot  runs  are 
sometimes  used  to  estimate  the  truncation  point  to  prevent  any  chance  of  statistical  bias. 
However,  Law  and  Kelton  suggest  that,  if  the  resulting  truncated  sequences  are  substantially 
larger  than  the  truncated  portion,  using  the  same  runs  for  truncation-point  identification 
and  confidence  interval  construction  is  “probably  safe”  [53:552].  The  same  runs  are  used  for 
both  selecting  the  truncation  points  and  making  the  confidence  intervals  in  the  Monte  Carlo 
analysis. 

6.3  Measures  of  Effectiveness 

The  performance  measures  are  different  than  single-run  approaches  because  different 
techniques  to  draw  inferences  are  applicable  with  multiple  replications.  In  this  study,  three 
of  the  measures  used  by  Kelton  and  Law  [44]  are  applied.  The  three  measures  are  the 
Point  Estimator  Bias  (PEBIAS),  the  ^'ean  Absolute  Deviation  (MAD),  and  the  method 
of  replication  coverage  rates  with  the  corresnonding  confidence  interval  half  widths.  These 
measures  are  applicable  to  model  outputs  for  which  the  steady-state  mean  can  be  calculated 
analytically. 

The  PEBIAS  measures  the  error  in  the  mean  estimated  with  the  truncated  sequences 
and  the  theoretical  mean.  From  each  truncated  sequence  of  length  N',  the  mean  estimate 
is  the  average  of  N'  observations.  The  estimated  mean  fiy  is  the  grand  mean  of  the 
M  truncated  sequences  Av  ~  m  Sm=i  ym{N'),  and  the  point  estimator  bias  is 

PEBIAS  =  fiy  —  fiy 

The  PEBIAS  is  presented  as  a  percentage  of  the  theoretical  mean  fiy,  . 

The  MAD  is  the  absolute  value  of  the  error  between  the  estimated  and  theoretical 

mean: 

MAD  =  |A„  -  Mvl 
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Therefore,  the  MAD  is  the  absolute  value  of  the  PEBIAS.  With  repeated  applications,  neg¬ 
ative  and  positive  values  of  PEBIAS  can  cancel  each  other.  In  contrast,  the  MAD  values  are 
all  positive. 

The  output  analysis  objective  is  to  make  inference  about  the  system.  These  inferences 
usually  are  stated  through  point  estimates  with  associated  confidence  intervals.  Therefore, 
the  coverage  rates  for  the  confidence  intervcds  and  their  associated  half  widths  are  impor¬ 
tant  evaluation  measures.  The  method  of  replications.  Equation  (3),  is  used  to  calculate 
a  confidence  interval  for  the  mean  of  the  truncated  sequence.  Along  with  these  perfor¬ 
mance  measures,  Schruben’s  initial  bias  test.  Equation  (10),  can  be  applied  to  the  truncated 
sequence  of  averaged  output.  Since  the  same  biased  variance  of  the  mean  estimate  as  Chap¬ 
ter  V  on  page  167  is  applied,  the  Schruben’s  test  statistic  is  questionable.  Except  for  the 
addition  of  Schruben’s  initial  bias  test,  these  are  the  same  performance  measures  that  Kelton 
and  Law  [44]  apply.  The  same  simulation  models  also  are  used  in  an  attempt  to  make  some 
performance  comparisons. 

6-4  Monte  Carlo  Results 

In  this  section,  several  analyses  are  conducted.  Kelton  and  Law’s  algorithm  [44]  re¬ 
quires  the  analyst  to  specify  nine  parameters.  Because  of  the  difficulty  in  determining  ap¬ 
propriate  parameter  values  and  implementing  their  algorithm,  a  Monte  Carlo  analysis  com¬ 
parable  to  their  results  is  conducted.  In  this  analysis,  the  scalar  k  is  set  to  1.0  as  suggested 
in  the  last  chapter.  In  the  next  Monte  Carlo  analysis,  the  estimated  value  of  k  is  used  and 
the  simulation  run  length  is  fixed.  These  results  include  evaluations  of  periodic  truncation 
points  for  sensitivity  analysis.  The  final  two  sections  present  exct  ‘on  on  the  effects  of 
simulation  run  length  and  MMAE  filter  spacing. 

Eleven  simulation  models  are  used  to  test  the  algorithm  in  the  first  Monte  Carlo 
analysis.  The  first  nine  are  queuing  systems  with  a  mean  interarrival  time  of  1.0.  Three 
simulations  of  M/M/1  queues  with  first-in-first-out  (FIFO)  queue  discipline,  empty  and  idle 
initial  conditions,  and  infinite  queue  capacity  are  used.  The  traffic  intensity  p  is  varied 
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between  0.8,  0.9,  and  0.95.  The  remainder  of  the  queuing  simulations  have  a  traffic  intensity 
p  of  0.8.  Two  variants  of  the  M/M/1  queue  are  changing  the  queue  discipline  to  last-in-first- 
out  (LIFO)  or  changing  the  initial  conditions  to  an  initial  queue  length  Lq  of  ten.  Another 
queuing  simulation  is  an  where  interarrival  times  are  a  four-stage  Erlang  random 

variable  [77:109].  The  number  of  servers  is  increased  to  two,  an  M/M/2  queue,  and  to  four, 
an  M/M/4  model.  The  final  queuing  simulation  is  three  M/M/1  queues  in  tandem  with  an 
interarrival  rate  at  each  queue  of  1.0  and  traffic  intensities  p  of  0.5,  0.7,  and  0.9.  (For  queues 
in  series  with  exponentially  distributed  interarrival  times  and  service  times,  the  interarrival 
rates  are  equal.)  The  queues  in  tandem  are  labeled  M/M/ 1 /M/M/ 1 /M/M/ 1. 

The  last  two  models  are  discrete-event  simulation  models  of  computer  systems  consid¬ 
ered  by  Law  and  Carson  [51].  The  time-sharing  computer  system  has  50  terminals  submitting 
jobs  to  a  FIFO  queue  at  the  CPU.  Each  computer  operator  “thinks”  for  an  exponentially 
distributed  time  with  a  mean  of  25.0  and  then  submits  a  job  requesting  an  exponentially 
distributed  service  time  with  mean  of  0.8.  The  CPU  processes  jobs  in  increments  of  0.1  time 
units  with  a  fixed  overhead  for  each  session  of  0.015.  Unfinished  jobs  enter  the  CPU  queue. 
Completed  jobs  return  to  the  terminal  and  the  operator  “thinks”  again.  The  output  is  the 
sequence  of  response  times  to  the  terminals.  The  time-sharing  computer  model’s  analytical 
steady-state  values  are  calculated  as  shown  by  Adiri  and  A vi- Itzhak  [3]. 

The  second  computer  simulation  has  eight  jobs  in  the  system.  CPU  service  times  are 
exponentially  distributed  with  a  mean  of  1.0  time  units.  After  leaving  the  CPU,  the  jobs 
enter  one  of  two  peripheral  queues  with  probability  of  0.5  each.  The  mean  service  times 
at  the  peripherals  are  exponentially  distributed  with  mean  of  2.0  time  units.  When  a  job 
completed  service  at  the  peripheral,  another  job  instantaneously  arrives  in  the  CPU  queue. 
Initial  conditions  are  one  peripheral  queue  with  six  jobs,  one  job  in  the  other  peripheral 
queue,  and  one  job  in  the  CPU  queue.  The  output  response  is  the  time  for  each  job  to  be 
processed  by  the  CPU  and  a  peripheral  device. 

The  theoretical  steady-state  output  response  times  for  these  models  are  shown  in  Ta¬ 
ble  70  [44].  The  Monte  Carlo  analysis  which  follows  is  divided  into  four  sections.  In  the  first 
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Table  70.  Theoretical  Steady-State  Means 


Model 

M/M/1  p  =  0.8 

M/M/1  p  =  0.9 

M/M/1  p  =  0.95 

■fapHti 

M/M/1  p  =  0.8  LIFO 

iSII 

M/M/1  p  =  0.8  Lo  =  10 

IBI 

EaIMII  p  =  0.8 

1.814 

M/M/2  p  =  0.8 

2.844 

M/M/4  p  =  0.8 

2.386 

M/M/ 1  /M/M/ 1  /M/M/ 1 

10.233 

Time- Sharing  Computer 

21.384 

Central  Server  Computer 

10.000 

section,  simulation  runs  are  made  under  the  same  conditions  as  Kelton  and  Law  [44].  The 
next  section  shows  runs  at  a  consistent  simulation  run  length.  Two  sections  of  analytical 
excursions  are  included;  one  section  examines  the  effect  of  increasing  the  simulation  run 
lengths,  and  the  other  section  examines  the  effect  of  the  filter  spacing. 

6.4-1  Results  with  ^  =  1.0  This  Monte  Carlo  analysis  is  conducted  in  order  to  com¬ 
pare  results  with  Kelton  and  Law  [44].  As  recommended  in  Chapter  V,  the  MMAE  approach 
is  implementd  with  k  arbitrarily  set  equal  to  one. 

Kelton  and  Law  [44]  propose  a  methodology  to  determine  the  appropriate  simulation 
run  length  N  and  truncation  point  no  for  replicated  simulations.  Their  algorithm  is  dis¬ 
cussed  on  page  2.4.2.  Their  algorithm  determines  the  run  length  by  restarting  all  of  the  M 
simulations  if  the  length  is  found  to  be  insufficient.  Restarting  M  replications  is  considered 
impractical  for  many  discrete-event  simulation  applications.  Because  of  the  difficulty  in  set¬ 
ting  the  parameters  and  applying  Kelton  and  Law’s  algorithm  (See  discussion  beginning  on 
page  2.4.2.),  their  algorithm  is  not  implemented.  However,  in  order  to  make  comparisons 
on  as  equitable  a  basis  as  possible,  the  same  simulation  models  and  evaluation  mecisures  are 
used.  In  addition,  the  models  are  run  for  the  same  number  of  replications,  five,  and  for  the 
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Table  71.  Correlation  Technic 

[ue  Parameter 

Estimation  Summary  (k 

=  1.0) 

Data  Model 

y 

Ay 

- 

^y 

m 

<f>2 

AA _ 

k 

h 

M/M/l  p  =  0.80 

3.20 

3.21 

0.53 

|inj« 

1.00 

0.00 

M/M/l  p  =  0.90 

8.10 

8.10 

2.14 

1.00 

0.00 

M/M/l  p  =  0.95 

18.05 

17.30 

5.89 

1.00 

0.00 

M/M/l  p  =  0.80  LIFO 

WEH 

3.30 

0.63 

0.45 

0.15  1 

0.22 

0.16 

0.78 

0.17 

M/M/l  p  =  0.80  Lo  =  10 

m 

3.20 

0.58 

0.88 

0.06 
_ _ ] 

0.07 

0.05 

1.00 

0.00 

E^IMIl  p  =  0.80 

1.81 

1.82 

“OT 

0.86 

0.07 

0.07 

0.06 

1.00 

0.00 

M/M/2  p  =  0.80 

2.84 

2.86 

0.56 

0.69 

0.09 

0.27 

0.09 

0.97 

0.02 

M/M/4  p  =  0.80 

2.39 

2.40 

0.55 

0.64 

0.15 

0.34 

0.15 

0.89 

0.05 

M/M/ 1  /M/M/ 1  /M/M/ 1 

10.23 

10.19 

2.22 

0.89 

0.07 

0.09 

0.07 

0.99 

0.01 

Time-Sharing  Computer 

21.38 

21.39 

0.83 

0.15 

0.49 

0.29 

0.60 

0.25 

0.35 

Central  Server  Computer 

10.01 

0.18 

0.38 

0.18 

0.38 

0.23 

0.42 

0.17 

run  length  of  the  average  run  length  from  their  results. 

Another  difference  in  the  results  is  that  their  algorithm  always  estimates  a  truncation 
point.  For  their  results,  the  maximum  simulation  length  is  set  to  3,000  observations  and 
the  maximum  deletion  amount  of  one  half.  Since  1,500  appears  to  be  beyond  the  transient 
for  these  output  sequences,  their  algorithm  defaults  to  a  “good”  truncation  point.  Since  the 
MMAE  approach  seldom  fails  to  estimate  a  truncation  point,  this  difference  does  not  impact 
the  comparison  sigruii^antly. 

A  Monte  Carlo  aneilysis  is  conducted  with  1,000  sets  of  runs  with  5  replications  per  set. 
The  MMAE  parameters  are  estimated  using  the  last  half  of  the  averaged  output  sequence. 
Table  71  shows  the  average  parameter  estimates  and  their  standard  deviations.  The  average 
initial  mean  estimate  fiy  is  very  near  the  theoretical  steady-state  mean  value  fiy,  except  for  the 
M/M/l  model  with  traffic  intensity  p  of  0.95.  The  estimated  AR(2)  with  measurement  noise 
model  often  has  significant  first  autoregressive  coefficients  (f>i  with  small  second  autoregressive 
coefficients  <f)2  and  no  measurement  noise.  A:  =  1.0.  In  general,  these  models'  outputs  are 
adequately  represented  by  a  pure  AR(1)  model. 

Based  on  the  scalar  esimate  k  and  the  variance  of  the  residuals  for  the  last  half  of  the 
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Table  72.  Noise  Variances  Estimation  Summary  (k  =  1.0) 


Data  Model 

Qa 

R 

i^R 

<^R 

M/M/1  p  =  0.80 

0.23 

0.00 

0.00 

M/M/1  p  =  0.90 

0.32 

■ifIB 

0.00 

0.00 

M/M/1  p  =  0.95 

0.39 

0.06 

0.00 

0.00 

M/M/1  p  =  0.80  LIFO 

13.57 

10.32 

4.14 

6.12 

M/M/1  p  =  0.80  Lq  =  10 

0.23 

0.02 

0.00 

E^IM/l  p  =  0.80 

0.13 

0.01 

0.00 

0.00 

M/M/2  p  =  0.80 

0.31 

0.03 

0.01 

0.01 

M/M/4  p  =  0.80 

0.38 

0.07 

0.05 

0.02 

M/M/l/M/M/l/M/M/1 

■Gi 

0.00 

Time-Sharing  Computer 

27.67 

63.95 

30.21 

Central  Server  Computer 

1.69 

1.11 

1.01 

averaged  output  sequence,  the  dynamics  noise  variance  Qj  and  the  measurement  noise  vari¬ 
ance  R  are  estimated.  The  estimation  approach  and  its  problems  are  discussed  on  pages  90 
and  167.  Table  72  shows  the  average  estimates  for  the  1,000  Monte  Carlo  sets  of  runs. 
Whenever  k  =  1.0,  the  measurement  noise  variance  R  is  also  estimated  to  be  zero. 

Table  73  shows  statistics  on  the  selected  truncation  points  ho  for  the  1,000  Monte  Carlo 
sets  of  runs.  The  column  labeled  ‘Fail’  indicates  the  proportion  of  sets  for  which  the  MMAE 
approach  failed  to  select  a  truncation  point.  These  sets  of  runs  are  excluded  from  further 
analysis.  The  averages  of  the  MMAE  selected  truncation  points  are  always  less  than  ^  of 
the  average  truncation  points  reported  by  Kelton  and  Law  [44].  For  the  E4/A//I  model,  the 
average  is  only  |  of  the  average  they  report.  Kelton  and  Law  report  the  accuracy  for  their 
estimation  of  the  truncation  point  tiq.  Using  this  accuracy  value  to  back  out  the  variance,  the 
MMAE  truncation-point  variance  is  less  for  every  model  except  M/M/4.  For  the  M/M/4, 
the  truncation-point  variances  are  almost  equal.  Therefore,  the  MMAE  algorithm  generally 
truncates  less  of  the  initial  data  than  Kelton  and  Law’s  algorithm.  If  the  MMAE  truncation 
points  are  sufficient  to  remove  the  initial  data  bias,  the  resulting  inferences  are  more  precise 
because  of  the  increased  number  of  untruncated  observations. 
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Table  73.  Truncation  Points  fip  {k  =  1.0) 


Data  Model 

Run  Length 

Fail 

Min 

Max 

Avg 

St  Dev 

M/M/1  p  =  0.80 

1615 

0.000 

1 

1422 

178.3 

184.9 

M/M/1  p  =  0.90 

1910 

0.013 

11 

1860 

443.3 

367.3 

M/M/1  p  =  0.95 

2560 

0.256 

18 

2424 

667.1 

458.7 

M/M/1  p  =  0.80  LIFO 

28 

1360 

335.3 

286.6 

M/M/1  p  =  0.80  LIFOt 

24 

1360 

291.6 

261.4 

M/M/1  p  =  0.80  Lo  =  10 

1475 

0.001 

1 

1373 

121.0 

154.5 

E4/M/I  p  =  0.80 

UBESm 

0.000 

1 

1253 

93.8 

114.3 

M/M/2  p  =  0.80 

0.009 

3 

1434 

308.1 

309.2 

M/M/4  p  =  0.80 

1390 

_ J 

0.489 

8 

1372 

486.0 

376.7 

M/M/1  /M/M/l/M/M/1 

0.004 

23 

1847 

558.4 

437.9 

Time-Sharing  Computer 

0.366 

31 

1375 

125.3 

142.9 

Time-Sharing  Computer  f 

1380 

0.109 

37 

1324 

122.6 

166.9 

Central  Server  Computer 

1150 

0.128 

8 

1110 

89.4 

148.9 

t  Truncation  algorithm  used  the  estimated  k  rather  than  k  =  1.0. 


The  simulation  models  for  which  the  average  of  k,  fij^,  is  less  than  one,  shown  in 
Table  71,  also  result  in  a  high  proportion  of  sets  where  MMAE  failed  to  identify  a  truncation 
point.  Because  of  the  analysis  of  k  in  Chapter  V,  the  MMAE  algorithm  is  implemented  with 
k  set  arbitrarily  to  one.  For  the  two  models  with  the  lowest  average  for  k,  M/M/1  LIFO  and 
Time-Sharing  Computer,  the  Monte  Carlo  analysis  is  run  again  with  the  MMAE  algorithm 
modified  to  use  the  estimated  k  rather  than  setting  it  to  one.  The  results  with  the  estimate 
k,  marked  with  the  dagger  f,  show  that  this  change  reduces  the  rate  of  failures  with  only  a 
small  effect  on  the  average  selected  truncation  points.  This  change  to  using  the  estimated 
k  in  the  algorithm  has  little  or  no  effect  on  most  of  the  cases  since  their  estimates  of  k  are 
close  to  1.0. 

Table  74  shows  the  measures  of  effectiveness  for  each  of  the  tested  models.  The  PEBIAS 
are  larger  than  values  reported  by  Kelton  and  Law  [44]  except  for  the  E^/M/l  and  Central 
Server  Computer  models.  Larger  errors  indicate  worse  results,  but  the  impact  is  mitigated 
by  two  considerations.  First,  the  percentages  of  PEBIAS  are  reported  in  terms  of  the  steady- 
state  means  fiy.  Since  these  percentages  are  not  large,  the  PEBIAS  values  are  very  small  in 
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Table  74.  Truncation  Point  rio  Evaluations  {k  =  1.0) 


Data  Model 

PEBIAS 

(%  of  Py) 

MAD 

Cov 

Half  Width 

Bias 

Test 

M/M/l  p  =  0.80 

2.4  ±  0.79 

0.3  ±  0.02 

0.8  ±  0.03 

0.28 

M/M/l  p  =  0.90 

2.8  ±  1.37 

1.6  ±  0.07 

3.5  ±  0.15 

0.54 

M/M/l  p  =  0.95 

-14.2  ±  1.27 

3.9  ±  0.14 

7.2  ±  0.27 

0.55 

M/M/l  p  =  0.80  LIFO 

8.2  ±  2.44 

0.6  ±  0.07 

0.39 

M/M/l  p  =  0.80  LIFO  t 

6.3  ±  2.11 

0.5  ±  0.06 

0.36 

M/M/l  p  =  0.80  lo  =  10 

1.8  ±  0.94 

0.4  ±  0.02 

0.21 

Ei/M/1  p  =  0.80 

1.5  ±  0.78 

0.2  ±  0.01 

0.16 

M/M/2  p  =  0.80 

6.4  ±  1.44 

0.5  ±  0.03 

0.56 

M/M/4  p  =  0.80 

6.2  ±  2.34 

0.5  ±  0.05 

0.74 

M/M/l/M/M/l/M/M/1 

5.1  ±  1.33 

0.85  ±  0.018 

0.61 

Time-Sharing  Computer 

0.29 

Time-Sharing  Computer  f 

0.89  ±  0.017 

0.39 

Central  Server  Computer 

0.90  ±  0.017 

0.28 

t  Truncation  algorithm  used  the  estimated  k  rather  than  k  =  1.0. 


absolute  terms.  In  fact,  the  difference  may  be  accounted  for  by  the  data  variation  between 
the  two  separate  Monte  Carlo  analyses.  The  second  consideration  is  that  if  the  transient  data 
biases  the  mean  estimates,  the  PEBIAS  has  a  sign  indicative  of  the  upward  or  downward  bias 
induced  by  the  initial  conditions.  Only  two  of  the  cases,  the  Af/M/1  models  with  p  =  0.95 
and  Lq  =  10,  have  the  same  sign  of  PEBIAS  as  is  induced  by  the  model’s  initial  conditions. 
Apparently,  the  MMAE  selected  truncation  points  generally  remove  the  bias  induced  by  the 
initial  conditions. 

Whereas  the  PEBIAS  allows  positive  and  negative  values  to  cancel,  the  absolute  value 
in  the  MAD  measure  prevents  errors  from  canceling  each  others  effect.  As  a  result,  the 
MAD  measure  reflects  both  the  accuracies  and  variances  of  the  mean  estimates.  The  MAD 
values  are  lower  than  the  corresponding  values  reported  by  Kelton  and  Law  [44]  except 
for  the  M/M/l  LIFO  and  the  A//A//1/A//M/1/M/A//1  models.  For  these  two  models, 
the  reported  MAD  values  are  within  0.1  of  their  results.  Therefore,  in  terms  of  MAD,  the 
MMAE  truncation  points  are  generally  better  than  the  points  selected  by  Kelton  and  Law’s 
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technique. 

The  coverage  rates  for  confidence  intervals  based  on  the  sequences  truncated  at  the 
MMAE  selected  truncation  points  are  higher  than  reported  by  Kelton  and  Law  [44]  except 
for  the  Af/M/l  with  p  =  0.95  model.  In  that  case,  the  coverage  rates  are  statistically 
indistinquishable,  0.69  and  0.70.  Thus,  in  terms  of  coverage  rates,  the  MMAE  truncation 
points  are  superior.  However,  coverage  rates  should  be  examined  in  conjunction  with  their 
corresponding  average  confidence  interval  average  half  width.  The  average  half  widths  are 
less  except  for  three  models.  For  the  M/M/l  LIFO,  M/M/2,  and  M/M/4:  models,  the 
average  half  widths  are  only  0.1  larger  for  the  sequence  truncated  with  the  MMAE  selected 
truncation  points.  The  higher  coverage  with  generally  smaller  average  half  width  results  from 
an  increase  in  the  number  of  steady-state  observations  since  the  MMAE  selected  truncation 
points  are  generally  much  earlier  in  the  output  sequence. 

Overall,  the  MMAE  selected  truncation  points  appeax  superior  to  those  selected  by 
Kelton  and  Law’s  algorithm.  The  results  are  not  conclusive  since  Kelton  and  Law’s  algorithm 
is  not  actually  implemented.  However,  for  this  Monte  Carlo  analysis,  which  uses  conditions 
as  similar  as  possible  to  their  Monte  Carlo  analysis,  better  results  are  obtained  by  the  MMAE 
truncation-point  selection  algorithm. 

6.4-2  Results  with  Estimated  k.  As  shown  in  Table  74,  using  the  estimated  k  generally 
improves  the  evaluations  of  the  truncated  sequence.  Since  better  results  are  obtained  when 
the  estimated  k  is  used  in  the  MMAE  selection  algorithm,  the  Monte  Carlo  analysis  is 
conducted  again.  Besides  using  the  estimate  of  fc,  the  simulation  run  length  is  fixed  at  1,500 
observations  for  each  of  the  5  replications. 

One  additional  output  sequence  for  the  M/M/l  queue  also  is  tested.  The  sequence 
consists  of  the  length  of  the  queue  L,  sampled  at  intervals  of  one  time  unit.  This  output 
sequence  for  the  model  M/M/l  (p  =  0.8  L,)  is  an  example  of  “statistics  based  on  time- 
persistent  variables”  in  contrast  to  all  the  other  tested  sequences,  which  are  “statistics  based 
on  observations”.  (See  discussion  of  output  classifications  on  page  37.) 
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Table  75.  Correlation  Technique  Parameter  Estimation  Summary  (Estimated  k) 


Data  Model 

y 

As 

■ 

IH 

(j)2 

^ 

H 

m 

M/M/1  p  =  0.80 

3.23 

0.56 

0.88 

0.06 

0.07 

0.05 

1.00 

0.00 

M/M/1  p  =  0.90 

8.15 

2.33 

0.91 

0.06 

0.07 

0.06 

1.00 

0.00 

M/M/1  p  =  0.95 

16.44 

5.53 

0.91 

0.06 

0.07 

0.06 

1.00 

0.00 

M/M/1  p  =  0.80  LIFO 

.20 

3.31 

0.59 

0.46 

0.15 

0.16 

M/M/1  p  =  0.80  Lq  =  10 

3.20 

3.20 

0.56 

0.88 

0.06 

IjB 

0.00 

M/M/1  p  =  0.80  I, 

3.20 

3.20 

0.59 

0.88 

0.06 

IfiW 

■mill 

Ei/M/l  p  =  0.80 

1.81 

1.82 

0.27 

0.86 

0.07 

1.00 

M/M/2  p  =  0.80 

2.84 

2.87 

0.56 

0.70 

0.09 

0.97 

M/M/4  p  =  0.80 

2.39 

2.41 

0.54 

0.64 

0.15 

0.89 

M/M/l/M/M/l/M/M/1 

10.23 

10.25 

2.54 

0.08 

0.07 

EES 

0.01 

Time-Sharing  Computer 

21.38 

21.39 

0.84 

0.49 

0.61 

0.35 

Central  Server  Computer 

10.00 

10.00 

0.16 

0.17 

iltB 

0.22 

0.15 

Table  75  shows  a  summary  of  the  estimated  system  parameters.  Except  for  the  M/M/1 
with  p  =  0.95  model,  the  averge  of  the  mean  estimates  are  very  close  to  the  theoretical 
models.  The  M/M/1  with  p  =  0.95  results  are  discussed  later  in  Section  6.4.3.  Table  76 
presents  the  corresponding  estimates  of  the  dynamics  noise  variance  Qj  and  the  measurement 
noise  variance  R. 

Table  77  shows  the  truncation-point  statistics  resulting  with  the  MMAE  algorithm 
using  the  estimated  k.  Even  when  the  run  length  is  slightly  longer  than  the  Monte  Carlo 
analysis  with  k  fixed  at  1.0,  the  average  truncation  point  is  slightly  earlier  in  the  output 
sequence.  The  proportion  of  times  that  the  MMAE  algorithm  failed  to  find  a  truncation 
point  significantly  decreases,  except  for  the  output  from  M/M/1  with  p  =  0.95.  For  this 
model,  the  failure  rate  increases  because  the  run  length  is  reduced  from  2,560  observations 
to  only  1,500  output  values. 

Table  78  shows  the  evaluation  measures  for  the  modified  MMAE  algorithm.  The  slight 
changes  from  the  case  in  which  the  MMAE  used  k  =  1.0  are  a  result  of  different  run  lengths, 
different  stochastic  realizations,  and  slightly  earlier  truncation  points.  With  the  decrease  in 
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Table  76.  Noise  Variances  Estimation  Summary  (Estimated  k) 


Data  Model 


M/M/1  p  =  0.80 
M/M/1  p  =  0.90 
M/M/1  p  =  0.95 


M/M/1  p  =  0.80  LIFO 
M/M/1  p  =  0.80  Lo  =  10 
M/M/1  p  =  0.80  L, 


E4/M/I  p  =  0.80 
M/M/2  p  =  0.80 
M/M/4  =  0.80 


M/M/l/M/M/l /M/M/1 
Time-Shai^'n  Computer 
Central  S»’'7er  Computer 


Table  77.  Truncation  Points  ho  (Estimated  k 


Data  Model 


M/M/1  p  =  0.80 
M/M/1  p  =  0.90 
M/M/1  p  =  0.95 


M/M/1  p  =  0.80  LIFO 
M/M/1  p  =  0.80  Lo  =  10 
M/M/1  p  =  0.80  L, 


E^jMIl  p  =  0.80 
M/M/2  p  =  0.80 
M/M/4  p  =  0.80 


M/M/ 1 /M/M/l/M/M/l 
Time- Sharing  Computer 
Central  Server  Computer 


Run  Length 


5 
5 

1500 


0.23 

0.32 

0.40 


13.99 

0.23 

0.29 


0.13 

0.31 

0.37 


0.58 

29.18 

1.61 


Min 

Max 

1 

1317 

9 

1458 

40 

1440 

29 

1456 

1 

1221 

1 

1489 

1 

1169 

5 

1482 

14 

1493 

488.5 

118.8 

38.8 
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Table  78.  Truncation  Point  Up  Evaluations  (Estimated  fc) 


Data  Model 

PEBIAS 

MAD 

Cov 

Half  Width 

Bias 

(%  of  Py) 

Test 

M/M/l  p  =  0.80 

3.0  ±  0.94 

0.86  ±  0.018 

0.9  ±  0.03 

kiKin 

M/M/l  p  =  0.90 

3.4  ±  1.52 

0.83  ±  0.020 

3.9  ±  0.17 

M/M/l  p  =  0.95 

-22.1  ±  1.48 

0.56  ±  0.032 

M/M/l  p  =  0.80  LIFO 

5.0  ±  1.30 

0.4  ±  0.04 

0.86  ±  0.018 

1.0  ±  0.05 

0.32 

M/M/l  p  =  0.80  Lo  =  10 

1.1  ±  0.79 

0.4  ±  0.02 

0.87  ±  0.018 

0.8  ±  0.03 

0.21 

M/M/l  p  =  0.80  L, 

3.2  ±  1.03 

0.4  ±  0.02 

0.86  ±  0.018 

1.0  ±  0.05 

0.34 

E^IMIl  p  =  0.80 

1.0  ±  0.65 

0.2  ±  0.01 

0.89  ±  0.017 

0.15 

M/M/2  p  =  0.80 

6.4  ±  1.32 

0.4  ±  0.03 

0.88  ±  0.017 

0.56 

A//M/4  p  =  0.80 

8.3  ±  2.40 

0.86  ±  0.023 

0.76 

M/M/l/M/M/l/M/M/1 

4.6  ±  1.39 

2.0  ±  0.10 

0.84  ±  0.019 

4.3  ±  0.19 

0.66 

Time-Sharing  Computer 

-0.2  ±  0.17 

0.5  ±  0.02 

0.90  ±  0.017 

1.2  ±  0.03 

0.36 

Central  Server  Computer 

0.1  ±  0.06 

0.1  ±  0.00 

0.90  ±  0.016 

0.2  ±  0.00 

0.19 

run  length,  the  M/M/l  with  p  =  0.95  coverage  rate  decreases  from  0.69  to  only  0.56.  For 
this  case,  the  effect  of  increasing  the  run  length  is  examined  in  Section  6.4.3. 

The  following  sequence  of  tables  characterizes  the  transient  and  its  effect  for  each  of 
the  eleven  models.  For  each  model,  two  tables  are  presented.  The  first  table  shows  the 
average  values  of  periodic  output  values  and  their  variance.  When  the  average  approaches 
the  steady-state  value,  the  transient  is  essentially  complete.  These  values  are  obtained  by 
averaging  over  the  1,000  Monte  Carlo  sets  each  with  5  replications.  Table  79  shows  the 
results  for  the  output  of  the  M/M/l  queue  simulation  with  p  —  0.8.  The  initial  conditions 
of  an  empty  queue  and  idle  server  cause  the  first  customer  always  to  have  a  waiting  time  of 
zero.  The  downward  initial  bias  appears  to  have  a  very  small  effect  by  the  200th  observation 
since  their  average  of  3.16  is  almost  equal  to  the  theoretical  average  of  3.2.  Certainly,  the 
initial  bias  is  inconsequential  by  the  400th  observation.  The  double  horizontal  lines  indicate 
between  which  periodic  values  the  average  truncation  point  occurs.  For  the  M/M/l  with 
p  =  0.8  output,  the  average  truncation  point  is  179.9,  which  is  between  100  and  200. 

The  second  table  on  the  output  of  each  model  shows  the  effect  of  truncating  at  periodic 
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Table  79.  Periodic  Output  Values  Points  for  M/M/1  with  p  =  0.8  (py  =  3.2) 


n 

Avn 

1 

0.00 

0.00 

100 

2.99 

3.61 

200 

3.16 

3.93 

300 

3.17 

3.95 

400 

3.25 

3.93 

500 

3.19 

3.93 

600 

3.23 

4.01 

700 

3.20 

4.05 

800 

3.21 

3.87 

900 

3.19 

4.04 

1000 

3.21 

3.98 

1100 

3.22 

4.01 

1200 

3.22 

3.86 

1300 

3.21 

3.89 

1400 

3.22 

3.91 

1500 

3.19 

3.93 

points.  Since  no  alternative  analytical  truncation-point  technique  is  widely  accepted,  these 
tables  investigate  the  sensitivity  of  the  MMAE  selected  truncation  points.  In  the  table  for 
each  model,  the  first  two  columns  are  the  mean  estimate  and  its  variance  when  the  output 
sequences  are  truncated  at  that  point.  The  next  columns  present  the  point  estimator  bias 
(PEBIAS)  as  a  percentage  of  the  theoretical  steady-state  mean  and  the  average  of  the  mean 
absolute  deviation  (MAD)  for  the  truncated  sequences.  The  coverage  rate  and  average 
confidence  interval  half  width  are  also  presented  for  the  truncated  sequences.  The  final 
column  presents  the  ratio  of  averaged  sequences  which  passed  Schruben’s  test  for  no  initial 
bias.  Again,  the  interval  containing  the  average  of  the  MMAE  selected  truncation  points  is 
indicated  with  double  horizontal  lines. 

Table  80  shows  the  inferences  that  result  for  periodic  truncations  of  the  output  from 
the  M/M/1  model  with  p  =  0.8.  The  mean  estimate  after  deleting  the  first  100  observations 
has  no  significant  downward  bias  from  the  initial  conditions.  Deleting  more  of  the  initial 
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Table  80.  Periodic  Truncation  Points  for  M/M/1  with  p  =  0.8  (/Xy  =  3.2) 


mm 

PEBIAS  (%  of  py) 

MAD 

Cov 

HW 

Bias  Test 

0 

3.16 

0.40 

-1.2 

± 

0.65 

0.3 

± 

0.01 

0.86 

± 

0.018 

0.8 

± 

0.02 

100 

3.21 

0.42 

0.5 

± 

0.68 

0.3 

± 

0.01 

0.87 

± 

C.018 

0.8 

± 

0.02 

200 

3.22 

0.44 

0.6 

± 

0.71 

0.3 

± 

0.01 

0.87 

± 

0.018 

0.8 

± 

0.02 

0.35 

300 

3.22 

0.46 

0.7 

± 

0.74 

0.4 

± 

0.02 

0.87 

± 

0.018 

0.9 

± 

0.03 

0.50 

400 

3.23 

0.47 

0.8 

± 

0.77 

0.4 

± 

0.02 

0.86 

± 

0.018 

0.9 

± 

0.03 

0.63 

500 

3.23 

0.49 

0.9 

± 

0.80 

0.4 

i 

0.02 

0.88 

± 

0.017 

0.9 

± 

0.03 

0.74 

600 

3.23 

0.52 

0.8 

± 

0.84 

0.4 

± 

0.02 

0.88 

± 

0.017 

1.0 

± 

0.03 

0.85 

700 

3.23 

0.54 

0.9 

± 

0.88 

0.4 

± 

0.02 

0.88 

± 

0.017 

1.0 

± 

0.03 

0.92 

800 

3.22 

0.58 

0.7 

± 

0.95 

0.5 

i 

0.02 

0.86 

± 

0.018 

1.1 

± 

0.03 

0.95 

900 

3.22 

0.61 

0.6 

± 

0.99 

0.5 

± 

0.02 

0.85 

± 

0.018 

1.1 

± 

0.04 

0.97 

1000 

3.22 

0.65 

0.7 

± 

1.06 

0.5 

± 

0.02 

0.85 

± 

0.019 

1.2 

± 

0.04 

0.95 

1100 

3.21 

0.70 

0.5 

± 

1.14 

0.5 

± 

0.02 

0.86 

± 

0.018 

1.3 

± 

0.04 

0.90 

1200 

3.20 

0.79 

0.1 

± 

1.29 

0.6 

± 

0.03 

0.83 

± 

0.020 

1.4 

± 

0.05 

0.82 

1300 

3.19 

0.95 

-0.2 

i 

1.54 

0.7 

± 

0.03 

0.81 

i 

0.020 

1.7 

± 

0.06 

0.72 

1400 

3.22 

1.23 

0.5 

± 

1.99 

0.9 

± 

0.04 

0.81 

± 

0.021 

2.1 

0.08 

0.92 

observations  increases  the  variance  in  the  mean  estimate.  With  no  deletion,  no  =  0,  the 
PEBIAS  has  a  negative  sign  because  of  the  downward  bias  induced  by  the  initial  conditions. 
However,  after  deleting  only  100  observations,  the  PEBIAS  no  longer  shows  the  effects  of  the 
initial  conditions.  The  MAD  statistic  is  a  function  of  both  the  point  estimator  bias  and  its 
variance.  The  average  value  of  the  MAD  statistic  increases  steadily  as  more  data  is  deleted 
and  the  variance  of  the  point  estimator  Py  increases.  The  coverage  rate  never  attains  its 
nominal  value  of  0.9.  Besides  a  slight  increase  in  the  coverage  rate,  the  initial  transient  has 
almost  no  effect.  However,  deleting  more  than  half  the  data  causes  the  coverage  rate  to 
decrease.  The  average  confidence  interval  half  widths  steadily  increase  with  less  data.  The 
final  column  reports  the  percentage  of  truncated  averaged  sequences  which  passed  Schruben’s 
test  for  no  initial  bias.  The  nominal  rate  of  0.9  is  not  achieved  until  700  observations  are 
deleted.  Schruben’s  test  appears  overly  sensitive  to  potential  outliers.  Schruben’s  test  also 
results  in  odd  fluctuations  as  the  length  of  the  truncated  sequence  becomes  very  short. 

The  periodic  output  values  for  the  simulation  of  the  M/M/1  queue  with  p  =  0.9  are 
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Table  81.  Periodic  Output  Values  Points  for  MjMjl  with  p  =  0.9  {py  —  8.1) 


n 

^Vn 

1 

0.00 

0.00 

100 

5.63 

5.81 

200 

6.90 

7.18 

300 

7.40 

7.90 

400 

7.76 

8.27 

500 

7.83 

8.54 

600 

8.03 

8.78 

700 

7.96 

8.83 

800 

8.16 

9.01 

900 

8.16 

9.26 

1000 

8.16 

9.04 

1100 

8.11 

9.15 

1200 

8.19 

8.88 

1300 

8.12 

8.95 

1400 

8.13 

8.81 

1500 
_ 1 

8.05 

8.84 

shown  in  Table  81.  The  exponential  or  geometric  decay  of  the  transient  value  to  the  steady- 
state  mean  of  8.1  is  apparent.  The  effects  of  the  transient  appear  to  be  inconsequential  by 
the  800th  observations.  From  Table  82,  the  initial  transient  effects  on  the  mean  estimates 
are  seen  to  be  negligible  after  deleting  the  first  500  observations.  After  deleting  500  initial 
observations,  the  PEBIAS  no  longer  has  the  negative  sign  induced  by  the  start-up  condi¬ 
tions  and  the  coverage  rate  has  attained  its  maximum  value.  Deleting  further  observations 
increases  the  variance  of  the  mean  estimators,  which  causes  the  coverage  rate  to  decrease 
even  with  wider  confidence  intervals.  The  average  MMAE  selected  truncation  point  of  405.8 
appears  .  ery  reasonable. 

The  periodic  output  from  the  simulation  of  the  M/M/1  queue  with  p  —  0.95,  shown 
in  Table  83,  indicate  that  many  of  the  output  sequences  have  not  completed  their  initial 
transient  within  1,500  observations.  The  average  of  the  1,000  Monte  Carlo  sets,  with  5 
replications  each,  never  attains  the  steady-state  mean  of  18.05.  Therefore,  the  average  of 
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the  mean  estimate  based  on  the  last  half  of  the  data  y,  shown  in  Table  75,  is  significantly 
below  the  steady-state  mean. 

For  the  M/Mfl  with  p  =  0.95,  Table  84  shows  that  the  average  mean  estimate  also 
never  attains  the  theoretical  value.  Regardless  of  the  truncation  point,  a  downward  bias  is 
indicated  by  the  negative  sign  on  the  PEBIAS.  The  coverage  is  always  below  the  nominal 
rate  of  0.9,  but  increases  as  more  initial  data  is  truncated.  Because  many  of  these  output 
sequences  generally  continue  to  increase,  the  MMAE  truncation-point  selection  algorithm 
fails  to  pick  a  truncation  point  for  35.6  percent  of  the  sets. 

When  the  MMAE  algorithm  selected  a  truncation  point,  the  coverage  rate  is  only 
0.56.  This  is  lower  than  any  of  the  fixed  periodic  truncation  points,  probably  because  of  the 
manner  in  which  the  output  varies.  As  shown  in  Figure  2  in  Section  2.4.3,  MfMfl  with 
p  =  0.95  simulation  output  has  long  subsequences  of  thousands  of  observations  with  low 
average  values  and  occasional  subsequences  of  about  one  thousand  observations  with  very 


Table  82.  Periodic  Truncation  Points  for  M/M/1  with  p  =  0.9  {py  =  8.1) 


ho 

PEBIAS  (%  of  py) 

MAD 

Cov  HW 

Bias  Test 

0 

7.61  1.64 

-6.1  ±  1.05 

1.4  ±  0.05 

0.77  ±  0.022  2.9  ±  0.11 

100 

7.86  1.75 

-2.9  ±  1.12 

1.4  ±  0.06 

0.81  ±  0.020  3.1  ±  0.11 

BH 

200 

7.98  1.84 

-1.5  ±  1.19 

1.4  ±  0.06 

0.82  i  0.020  3.2  ±  0.12 

300 

8.04  1.94 

-0.7  ±  1.25 

1.5  ±  0.0/ 

0.83  ±  0.020  3.4  ±  0.13 

0.69 

400 

5.09  2.02 

-0.1  ±  1.30 

1.5  ±  0.07 

0.83  ±  0.020  3.5  ±  0.14 

0.70 

500 

8.12  2.10 

0.2  ±  1.35 

1.6  ±  0.07 

0.83  ±  0.020  3.7  ±  0.14 

0.75 

600 

8.13  2.19 

0.4  ±  1.41 

1.6  ±  0.08 

0.83  ±  0.020  3.9  ±  0.15 

0.75 

700 

8.15  2.27 

0.6  ±  1.46 

1.7  ±  0.08 

0.83  ±  0.020  4.0  ±  0.16 

0.83 

800 

8.15  2.39 

0.6  ±  1.53 

1.8  ±  0.08 

0.81  ±  0.020  4.2  ±  0.17 

0.82 

900 

8.14  2.48 

0.5  ±  1.59 

1.9  ±  0.08 

0.82  ±  0.020  4.3  ±  0.17 

0.81 

1000 

8.13  2.60 

0.4  ±  1.67 

2.0  ±  0.09 

0.81  ±  0.020  4.5  ±  0.18 

0.78 

1100 

8.12  2.75 

0.2  ±  1.77 

2.1  ±  0.09 

0.81  ±  0.021  4.8  ±  0.19 

0.78 

1200 

8.10  2.96 

0.0  ±  1.90 

2.3  ±  0.10 

0.79  ±  0.021  5.1  ±  0.19 

0.82 

1300 

8.06  3.18 

-0.5  ±  2.05 

2.5  ±  0.10 

0.78  ±  0.021  5.6  ±  0.20 

0.94 

1400 

8.07  3.49 

-0.4  ±  2.24 

2.8  ±  0.11 

0.78  ±  0.022  6.2  ±  0.21 

0.99 
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Table  83.  Periodic  Output  Values  Points  for  M/M/1  with  p  =  0.95  [py  =  18.05) 


n 

Avn 

^Vn 

1 

0.00 

0.00 

100 

7.61 

7.09 

200 

10.29 

9.48 

300 

11.86 

11.10 

400 

12.93 

12.24 

500 

13.73 

13.20 

600 

14.50 

14.00 

700 

14.94 

14.52 

800 

15.55 

15.26 

900 

15.89 

15.85 

1000 

16.14 

16.20 

1100 

16.47 

16.56 

1200 

16.73 

16.75 

1300 

16.90 

16.98 

1400 

17.01 

17.07 

1500 

17.01 

17.28 

Table  84.  Periodic  Truncation  Points  for  M/M/1  with  p  =  0.95  {py  =  18.05) 


no 

WBM 

PEBIAS  (%  of  py) 

MAD 

Cov 

HW 

Bias  Test 

14.05 

3.75 

-22.1  ±  1.08 

4.8  ±  0.14 

0.58  ±  0.026 

6.9  ±  0.23 

14.69 

4.00 

-18.6  ±  1.15 

4.5  ±  0.14 

0.63  ±  0.025 

7.4  ±  0.24 

15.13 

4.26 

-16.2  ±  1.23 

4.4  ±  0.14 

0.67  ±  0.024 

7.8  ±  0.26 

■H 

15.46 

4.51 

-14.4  ±  1.30 

4.3  ±  0.15 

0.69  ±  0.024 

8.3  ±  0.28 

■IB 

15.74 

4.75 

-12.8  ±  1.37 

4.4  ±  0.15 

0.70  ±  0.024 

8.7  ±  0.30 

0.47 

15.98 

4.98 

-11.5  ±  1.44 

4.5  ±  0.16 

0.72  ±  0.023 

9.2  ±  0.31 

0.50 

600 

16.19 

5.20 

-10.3  ±  1.50 

4.6  ±  0.16 

0.73  ±  0.023 

9.7  ±  0.33 

0.52 

700 

16.36 

5.41 

-9.3  ±  1.56 

4.7  ±  0.17 

0.73  ±  0.023 

10.1  ±  0.34 

0.78 

800 

16.51 

5.64 

-8.5  ±  1.63 

4.8  ±  0.17 

0.74  ±  0.023 

10.6  ±  0.36 

0.79 

900 

16.63 

5.84 

-7.9  ±  1.68 

4.9  ±  0.18 

0.77  ±  0.022 

11.0  ±  0.37 

0.77 

1000 

16.75 

6.03 

-7.2  ±  1.74 

5.0  ±  0.19 

0.77  ±  0.022 

11.4  ±  0.38 

0.81 

1100 

16.83 

6.23 

-6.8  ±  1.79 

5.1  ±  0.19 

0.77  ±  0.022 

11.9  ±  0.39 

0.87 

1200 

16.91 

6.48 

-6.3  ±  1.87 

5.3  ±  0.20 

0.77  ±  0.022 

12.4  ±  0.40 

0.93 

1300 

16.93 

6.78 

-6.2  ±  1.96 

5.5  ±  0.21 

0.78  ±  0.022 

13.0  ±  0.40 

0.98 

1400 

16.97 

7.07 

-6.0  ±  2.04 

5.8  ±  0.22 

0.80  ±  0.021 

13.8  ±  0.41 

0.99 
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Table  85.  Periodic  Output  Values  Points  for  Af/M/1  LIFO  {fiy  =  3.2) 


n 

Avn 

1 

0.00 

0.00 

100 

2.68 

7.21 

200 

3.07 

9.66 

300 

3.24 

10.71 

400 

3.45 

11.32 

500 

3.18 

10.21 

600 

3.10 

10.06 

700 

3.19 

10.62 

800 

3.57 

12.89 

900 

3.22 

10.52 

1000 

2.98 

9.69 

1100 

3.06 

9.84 

1200 

3.22 

11.20 

1300 

3.22 

11.21 

1400 

3.29 

10.77 

1500 

15.86 

28.97 

long  waiting  times  when  the  system  is  congested.  If  none  of  the  five  replications  happen  to 
have  become  congested  within  1,500  observations,  the  mean  estimate  based  on  the  last  half 
of  the  data  sequence  y  is  significantly  low.  As  a  result,  the  MMAE  mean  estimate 
very  likely  comes  within  1.0  of  y  and  a  truncation  point  is  selected.  Under  these  conditions, 
the  mean  estimate  and  confidence  interval  are  biased  downward  and  fail  to  cover  the  steady- 
state  mecin.  This  phenonmenon  explains  the  lower  coverage  rate  for  the  MMAE  truncation 
sequences  compared  to  the  sequences  truncated  at  fix  intervals. 

The  M/M/1  LIFO  average  observations,  shown  in  Table  85,  indicate  that  the  signif¬ 
icant  effects  of  the  transient  are  negligible  after  300th  observations.  From  Table  86,  the 
truncation  point  apparently  can  be  selected  earlier  in  the  output  sequence  without  adverse 
effects. 

The  output  from  the  simulation  of  the  M/M/1  with  initial  queue  length  Lq  of  10  is 
an  example  of  an  initially  over- congested  system.  Except  for  the  first  waiting  time,  which  is 
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still  zero,  the  entities  during  the  initial  transient  have  waiting  times  generally  longer  than 
the  steady-state  mean.  Table  87  shows  that,  by  the  300th  observation,  the  average  waiting 
time  has  decreased  to  less  than  the  steady-state  value.  Table  88  indicates  that  no  or  very 
little  truncation  achieves  the  highest  coverage  rate  with  the  smallest  average  confidence 
interval  half  widths.  The  MMAE  algorithm  generally  selects  very  early  truncation  points 
and  achieves  good  results. 

The  length  of  queue  L,  is  an  example  of  “statistics  based  on  time-persistent  variable.” 
The  output  M/M/1  p  =  0.8  Lq  is  generated  by  sampling  the  queue  length  every  time  unit. 
Therefore,  unlike  for  the  other  tested  models,  the  state  transition  matrix  $  represents  the 
change  over  a  fixed  time  interval  rather  than  the  change  between  sequential  observations. 
Table  89  indicates  that  the  output  is  very  near  its  steady-state  mean  by  the  200th  observation 
and  the  initial  trajisient  effectively  is  complete  by  the  400th  observation.  From  Table  90, 
the  effect  of  the  initial  conditions  is  insignificant  on  coverage  after  100  observations.  The 


Table  86.  Periodic  Truncation  Points  for  M/M/1  LIFO  {py  =  3.2) 


flQ 

Av 

wm 

PEBIAS  (%  of  py) 

MAD 

Cov 

HW 

Bias  Test 

3.15 

0.39 

-1.4 

± 

0.64 

0.3 

± 

0.01 

0.85 

± 

0.019 

0.7 

± 

0.02 

HIM 

Ira 

3.24 

0.42 

1.2 

± 

0.68 

0.3 

± 

0.01 

0.88 

± 

0.017 

0.8 

± 

0.02 

B9 

3.26 

0.43 

1.8 

± 

0.71 

0.3 

± 

0.01 

0.88 

± 

0.017 

0.8 

± 

0.02 

■9 

3.27 

0.46 

2.1 

0.75 

0.4 

± 

0.02 

0.88 

± 

0.017 

0.9 

± 

0.03 

3.27 

0.49 

2.3 

T 

0.79 

0.4 

± 

0.02 

0.87 

± 

0.017 

0.9 

± 

0.03 

0.46 

3.28 

0.51 

2.5 

± 

0.82 

0.4 

± 

0.02 

0.88 

± 

0.017 

1.0 

± 

0.03 

0.59 

3.29 

0.53 

2.7 

± 

0.87 

0.4 

± 

0.02 

0.88 

± 

0.017 

1.0 

± 

0.03 

0.70 

rainl 

0.57 

i 

0.4 

± 

0.02 

0.88 

± 

0.017 

1.1 

± 

0.04 

0.76 

800 

3.31 

0.62 

3.4 

± 

0.5 

± 

0.02 

0.87 

± 

0.018 

1.1 

± 

0.04 

0.84 

mm 

3.32 

0.67 

3.8 

± 

0.5 

± 

0.02 

0.87 

± 

0.017 

1.2 

± 

0.04 

0.88 

3.34 

0.72 

4.5 

i 

1.17 

0.5 

± 

0.03 

0.87 

i 

0.017 

1.3 

± 

0.04 

0.93 

1100 

3.38 

0.79 

5.6 

± 

1.28 

0.6 

± 

0.03 

0.87 

± 

0.017 

1.4 

± 

0.05 

0.96 

1200 

3.41 

6.7 

i 

1.47 

0.7 

± 

0.03 

0.86 

± 

0.018 

1.6 

± 

0.05 

0.97 

1300 

3.49 

1.13 

9.1 

± 

1.84 

0.8 

± 

0.04 

0.87 

± 

0.017 

1.9 

± 

0.07 

0.99 

1400 

3.79 

1.73 

18.4 

± 

2.82 

1.2 

± 

0.07 

0.86 

± 

0.018 

2.7 

± 

0.12 

0.99 
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Table  87.  Periodic  Output  Values  Points  for  M/M/1  with  Lq  =  10  (/Xy  =  3.2) 


n 

Ml/n 

^Vn 

1 

0.00 

0.00 

100 

3.36 

4.18 

200 

3.28 

4.07 

300 

3.15 

3.97 

400 

3.24 

3.86 

500 

3.26 

3.97 

600 

3.18 

3.88 

700 

3.23 

3.86 

800 

3.18 

3.90 

900 

3.19 

4.02 

1000 

3.29 

4.05 

1100 

3.25 

4.01 

1200 

3.21 

3.93 

1300 

3.22 

3.93 

1400 

3.20 

3.86 

1500 

3.09 

3.77 

Table  88.  Periodic  Truncation  Points  for  MfMjl  with  Lq  —  10  (/Xy  =  3.2) 


ho 

Av 

PEBIAS  (%  of  fiy) 

MAD 

Cov 

HW 

Bias  Test 

0 

3.29 

0.42 

2.9  ±  0.68 

0.3  ±  0.01 

0.88  ±  0.017 

0.8  ±  0.02 

100 

3.21 

0.43 

0.4  ±  0.69 

0.3  ±  0.01 

0.88  ±  0.017 

0.8  ±  0.02 

200 

3.20 

0.44 

0.2  ±  0.71 

0.3  ±  0.01 

0.87  ±  0.017 

0.8  ±  0.02 

300 

3.20 

0.45 

0.1  ±  0.74 

0.4  ±  0.01 

0.87  ±  0.018 

0.8  ±  0.02 

■H 

400 

3.21 

0.47 

0.2  ±  0.76 

0.4  ±  0.02 

0.87  ±  0.018 

0.9  ±  0.02 

0.65 

500 

3.20 

0.49 

0.1  ±  0.79 

0.4  ±  0.02 

0.87  ±  0.017 

0.9  ±  0.03 

0.77 

600 

3.20 

0.51 

0.0  ±  0.84 

0.4  ±  0.02 

0.87  ±  0.017 

0.9  ±  0.03 

0.83 

700 

3.20 

0.54 

0.1  ±  0.88 

0.4  ±  0.02 

0.87  ±  0.018 

1.0  ±  0.03 

0.92 

800 

3.20 

0.58 

0.0  ±  0.94 

0.4  ±  0.02 

0.86  ±  0.018 

1.1  ±  0.03 

0.96 

900 

3.20 

0.61 

0.0  ±  1.00 

0.5  ±  0.02 

0.86  ±  0.018 

1.1  ±  0.04 

0.97 

1000 

3.19 

0.65 

-0.2  ±  1.06 

0.5  ±  0.02 

0.86  ±  0.018 

1.2  ±  0.04 

0.96 

1100 

3.18 

0.70 

-0.7  ±  1.13 

0.5  ±  0.02 

0.84  ±  0.019 

1.3  ±  0.04 

0.91 

1200 

3.18 

0.78 

-0.6  ±  1.28 

0.6  ±  0.03 

0.83  ±  0.020 

1.4  ±  0.05 

0.82 

1300 

3.18 

0.87 

-0.7  ±  1.42 

0.7  ±  0.03 

0.81  ±  0.020 

1.6  ±  0.06 

0.71 

1400 

3.14 

_ 

1.10 

-1.7  ±  1.80 

0.9  ±  0.04 

0.79  ±  0.021 

1.9  ±  0.07 

0.94 
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average  MMAE  selected  truncation  point  is  207.1.  For  this  example  of  “statistics  based  on 
time-persistent  variables”,  the  MMAE  selected  truncation  point  is  both  effective  and  efficient 
at  reducing  the  effects  of  the  initial  conditions. 

For  the  E4/M/I  model,  the  periodic  average  output  values,  Table  91,  and  the  periodic 
truncation  points.  Table  92,  show  that  the  significant  transient  effects  are  relatively  short¬ 
lived.  As  such,  the  average  truncation  point  selected  by  the  MMAE  algorithm  is  only  91.7. 


For  the  Af/M/2  simulation  output,  the  averages  of  periodic  observations  and  the  eval¬ 
uation  of  periodic  truncation  points  are  shown  in  Tables  93  and  94,  respectively.  Since  the 
average  truncation  point  selected  by  the  MMAE  algorithm  is  300.3,  this  case  provides  the 
opportunity  to  examine  the  effect  of  the  variance  in  the  MMAE  truncation  points  compared 
to  a  fixed  truncation  point  of  300.  Since  the  evaluations  of  the  MMAE  truncation  points 
use  the  same  output  sequences  to  select  the  truncation  point  and  to  draw  inferences,  a  bias 
may  be  present.  For  the  1,000  sets  of  runs,  the  percentage  of  PEBIAS  is  6.4  for  the  MMAE 
approach  compared  to  0.8  the  fixed  truncation  point  of  300.  However,  6.4  percent  of  is 
only  0.18  in  absolute  terms.  This  is  rather  small  compared  to  the  standard  deviation  of  the 
mean  estimates  of  0.45.  The  average  MAD  values  are  about  the  same.  The  sequences 
truncated  with  the  MMAE  selected  points  have  a  slightly  larger  half  width  and  therefore 
a  coverage  rate  a  little  closer  to  the  nominal  rate.  The  largest  difference  is  in  the  rate  of 
passing  Schruben’s  test  for  initial  bias.  The  sequences  truncated  at  the  fix  point  pass  75 
percent  of  the  time,  while  only  56  percent  of  the  sequences  truncated  at  the  MMAE  selected 
point  pass  Schruben’s  test. 

Tables  95  and  96  indicate  that  the  output  from  the  M/MjA  model  has  a  relatively 
short  transient.  The  sequences  truncated  at  the  MMAE  selected  truncation  point  achieve 
a  coverage  of  0.87,  as  high  as  any  of  the  fixed  truncation  points.  While  the  statistics  and 
parameters  for  these  sequences  appear  similar  to  the  other  models,  the  MMAE  algorithm 
fails  to  select  a  truncation  point  in  37.2  percent  of  the  Monte  Carlo  sets  of  runs.  These 
failures  are  caused  by  the  filter  spacing,  which  is  examined  later  in  Section  6.4.4. 
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Table  89.  Periodic  Output  Values  Points  for  M/M/1  Lq  {fiy  =  3.2) 


n 

0.47 

0.82 

mi 

2.87 

3.80 

■a 

3.15 

4.12 

300 

3.17 

4.32 

400 

3.21 

4.28 

500 

3.23 

4.29 

600 

3.30 

4.43 

700 

3.23 

4.42 

800 

3.23 

4.39 

900 

3.17 

4.18 

1000 

3.17 

4.27 

1100 

3.22 

4.29 

1200 

3.17 

4.25 

1300 

3.19 

4.32 

1400 

3.21 

4.33 

1500 

3.09 

4.22 

Table  90.  Periodic  Truncation  Points  for  MIMjl  Lq  {fiy  =  3.2) 


no 

IKfll 

PEBIAS  (%  of  ny) 

MAD 

Cov 

HW 

Bias  Test 

0 

3.15 

0.41 

-1.6  ±  0.67 

0.3  ±  0.01 

0.84  ±  0.019 

0.8  ±  0.02 

WSSSM 

■sa 

3.20 

0.44 

-0.1  ±  0.71 

0.3  ±  0.01 

0.86  ±  0.018 

0.8  ±  0.02 

■rh 

■i 

3.21 

0.45 

0.3  ±  0.73 

0.4  ±  0.01 

0.86  ±  0.018 

0.8  ±  0.02 

■B 

300 

3.21 

0.47 

0.4  ±  0.77 

0.4  ±  0.02 

0.85  ±  0.019 

0.9  ±  0.02 

0.49 

400 

3.21 

0.50 

0.4  ±  0.82 

0.4  ±  0.02 

0.85  ±  0.019 

0.9  ±  0.03 

0.64 

500 

3.22 

0.52 

0.5  ±  0.85 

0.4  ±  0.02 

0.86  ±  0.018 

1.0  ±  0.03 

0.77 

600 

3.21 

0.54 

0.3  ±  0.88 

0.4  ±  0.02 

0.85  ±  0.018 

1.0  ±  0.03 

0.85 

700 

3.20 

0.57 

0.1  ±  0.92 

0.5  ±  0.02 

0.86  ±  0.018 

1.1  ±  0.03 

0.90 

800 

3.19 

0.61 

-0.3  ±  0.99 

0.5  ±  0.02 

0.85  ±  0.019 

1.1  ±  0.03 

0.94 

900 

3.20 

0.66 

-0.1  ±  1.07 

0.5  ±  0.02 

0.84  ±  0.019 

1.2  ±  0.04 

0.96 

1000 

3.21 

0.71 

0.4  ±  1.15 

0.5  ±  0.02 

0.84  ±  0.019 

1.3  ±  0.04 

0.96 

1100 

3.23 

0.80 

0.8  ±  1.30 

0.6  ±  0.03 

0.83  ±  0.019 

1.4  ±  0.05 

0.92 

1200 

3.21 

0.90 

0.3  ±  1.46 

0.7  ±  0.03 

0.81  ±  0.020 

1.6  ±  0.06 

0.85 

1300 

3.22 

1.07 

0.5  ±  1.74 

0.8  ±  0.04 

0.81  ±  0.021 

1.8  ±  0.07 

0.74 

1400 

3.20 

1.28 

0.1  ±  2.08 

1.0  ±  0.04 

0.79  ±  9.021 

2.2  ±  0.09 

0.89 
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Table  91.  Periodic  Output  Values  Points  for  E^jMIl  (fiy  =  1.814) 


n 

— - 

^Vn 

1 

0.00 

0.00 

100 

1.73 

2.38 

200 

1.82 

2.53 

300 

1.80 

2.52 

400 

1.87 

2.54 

500 

1.80 

2.45 

600 

1.82 

2.55 

700 

1.82 

2.55 

800 

1.81 

2.53 

900 

1.78 

2.50 

1000 

1.81 

2.47 

1100 

1.82 

2.49 

1200 

1.85 

2.51 

1300 

1.85 

2.51 

1400 

1.82 

2.50 

1500 

1.75 

2.46 

Table  92.  Periodic  Truncation  Points  for  E^IMjl  [fiy  =  1.814) 


no 

mm 

PEBIAS  (%  of  (iy) 

MAD 

Cov 

HW 

Bias  Test 

-1.0  ±  0.57 

0.2  ±  0.01 

0.88  ±  0.017 

0.4  ±  0.01 

0.00 

100 

1.82 

0.21 

0.2  ±  0.59 

0.2  ±  0.01 

0.89  ±  0.017 

0.4  ±  0.01 

0.15 

200 

1.82 

0.21 

0.1  ±  0.61 

0.2  ±  0.01 

0.88  ±  0.017 

0.4  ±  0.01 

0.34 

300 

1.82 

0.22 

0.2  ±  0.64 

0.2  ±  0.01 

0.89  ±  0.016 

0.4  ±  0.01 

0.47 

400 

1.82 

0.23 

0.2  ±  0.66 

0.2  ±  0.01 

0.89  ±  0.016 

0.4  ±  0.01 

0.61 

500 

1.82 

0.24 

0.2  ±  0.69 

0.2  ±  0.01 

0.89  ±  0.016 

0.5  ±  0.01 

0.73 

600 

1.82 

0.25 

0.2  ±  0.72 

0.2  ±  0.01 

0.88  ±  0.017 

0.5  ±  0.01 

0.81 

700 

1.82 

0.26 

0.4  ±  0.76 

0.2  ±  0.01 

0.89  ±  0.016 

0.5  ±  0.01 

0.88 

800 

1.82 

vm 

0.1  ±  0.81 

0.2  ±  0.01 

0.88  ±  0.017 

0.5  ±  0.02 

0.93 

900 

1.81 

0.0  ±  0.87 

0.2  ±  0.01 

0.87  ±  0.017 

0.6  ±  0.02 

0.97 

1.82 

0.2  ±  0.95 

0.3  ±  0.01 

0.87  ±  0.018 

0.6  ±  0.02 

0.98 

1100 

1.81 

-0.1  ±  1.06 

0.3  ±  0.01 

0.86  ±  0.018 

0.7  ±  0.02 

0.97 

1200 

1.82 

0.1  ±  1.22 

0.3  ±  0.01 

0.86  ±  0.018 

0.8  ±  0.03 

0.94 

1300 

1.81 

-0.2  ±  1.45 

0.4  ±  0.02 

0.83  ±  0.019 

0.9  ±  0.03 

0.82 

1400 

1.81 

-0.2  ±  1.91 

0.5  ±  0.02 

0.81  ±  0.021 

1.1  ±  0.05 

0.79 
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Table  93.  Periodic  Output  Values  Points  for  M/M/2  {fiy  —  2.844) 


n 

Avn 

1 

0.00 

0.00 

100 

2.64 

3.53 

200 

2.82 

3.84 

300 

2.85 

3.83 

400 

2.89 

3.84 

500 

2.85 

3.80 

600 

2.88 

3.89 

700 

2.87 

3.99 

800 

2.84 

3.79 

900 

2.82 

3.99 

1000 

2.86 

3.85 

1100 

2.88 

3.93 

1200 

2.88 

3.78 

1300 

2.85 

3.75 

1400 

2.87 

3.83 

1500 

2.81 

3.84 

Table  94.  Periodic  Truncation  Points  for  MJMll  {fiy  —  2.844) 


ho 

KM 

EM 

PEBIAS  (%  of  fly) 

MAD 

Cov 

HW 

Bias  Test 

0 

2.81 

0.39 

-1.4  ±  0.72 

0.3  ±  0.01 

0.86  ±  0.018 

0.7  ±  0.02 

■OBI 

100 

2.86 

0.41 

0.5  ±  0.76 

0.3  ±  0.01 

0.87  ±  0.018 

0.8  ±  0.02 

200 

2.86 

0.43 

0.7  ±  0.79 

0.3  zb  0.01 

0.86  ±  0.018 

0.8  ±  0.02 

■H 

300 

2.87 

0.45 

0.8  ±  0.82 

0.3  ±  0.02 

0.86  zb  0.018 

0.8  ±  0.03 

0.75 

400 

2.87 

0.47 

0.9  ±  0.86 

0.4  ±  0.02 

0.87  ±  0.018 

0.9  ±  0.03 

500 

2.87 

0.48 

1.0  ±  0.89 

0.4  ±  0.02 

0.88  ±  0.017 

0.9  ±  0.03 

■h 

600 

2.87 

0.51 

0.9  ±  0.93 

0.4  ±  0.02 

0.87  ±  0.017 

1.0  ±  0.03 

700 

2.87 

0.54 

1.0  ±  0.98 

0.4  zb  0.02 

0.87  ±  0.017 

1.0  zb  0.03 

0.97 

800 

2.86 

0.58 

0.7  zb  1.05 

0.4  zb  0.02 

0.86  ±  0.018 

1.0  ±  0.03 

0.96 

900 

2.86 

0.60 

0.6  ±  1.10 

0.5  ±  0.02 

0.85  ±  0.019 

1.1  ±  0.04 

0.96 

1000 

2.87 

0.64 

0.8  ±  1.17 

0.5  ±  0.02 

0.84  ±  0.019 

1.2  ±  0.04 

0.92 

1100 

2.86 

0.69 

0.5  ±  1.26 

0.5  ±  0.02 

0.85  ±  0.019 

1.3  ±  0.04 

0.87 

1200 

2.85 

0.78 

0.1  ±  1.42 

0.6  ±  0.03 

0.83  ±  0.020 

1.4  ±  0.05 

0.81 

1300 

2.84 

0.93 

-0.3  ±  1.70 

0.7  ±  0.03 

0.81  ±  0.020 

1.6  ±  0.06 

0.75 

1400 

2.86 

1.21 

0.5  zb  2.21 

0.9  ±  0.04 

0.81  ±  0.021 

2.0  ±  0.08 

0.95 
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Table  95.  Periodic  Output  Values  Points  for  M/M /A  =  2.386) 


n 

^Vn 

1 

0.00 

0.00 

100 

2.22 

3.39 

200 

2.37 

3.62 

300 

2.35 

3.65 

400 

2.42 

3.61 

500 

2.36 

3.62 

600 

2.41 

3.73 

700 

2.41 

3.77 

800 

2.40 

3.67 

900 

2.40 

3.80 

1000 

2.37 

3.71 

1100 

2.43 

3.77 

1200 

2.43 

3.61 

1300 

2.34 

3.54 

1400 

2.42 

3.67 

1500 

2.35 

3.69 

Table  96.  Periodic  Truncation  Points  for  M/M/4  [fiy  =  2.386) 


mm 

PEBIAS  (%  of  ^ly) 

MAD 

Cov 

HW 

Bias  Test 

0 

2.34 

0.38 

-1.8  ±  0.83 

0.3  ±  0.01 

0.85  ±  0.019 

0.7  ±  0.02 

mSm 

100 

2.40 

0.40 

0.4  ±  0.88 

0.3  ±  0.01 

0.86  ±  0.018 

0.8  ±  0.02 

200 

2.40 

0.42 

0.6  ±  0.91 

0.3  ±  0.01 

0.86  ±  0.018 

0.8  ±  0.02 

BIfl 

300 

2.40 

0.44 

0.8  ±  0.95 

0.3  ±  0.01 

0.86  ±  0.018 

0.8  ±  0.03 

400 

2.41 

0.46 

0.9  ±  0.99 

0.4  ±  0.01 

0.85  ±  0.018 

0.9  ±  0.03 

0.96 

500 

2.41 

0.47 

1.0  ±  1.03 

0.4  ±  0.02 

0.87  ±  0.017 

0.9  ±  0.03 

0.90 

600 

2.41 

0.49 

1.0  ±  1.08 

0.4  ±  0.02 

0.86  ±  0.018 

0.9  ±  0.03 

0.83 

700 

2.41 

0.52 

1.0  ±  1.14 

0.4  ±  0.02 

0.87  ±  0.018 

1.0  ±  0.03 

0.82 

800 

2.40 

0.56 

0.7  ±  1.22 

0.4  ±  0.02 

0.86  ±  0.018 

1.0  ±  0.03 

0.85 

900 

2.40 

0.58 

0.5  ±  1.27 

0.4  ±  0.02 

0.85  ±  0.019 

o 

o 

-H 

0.84 

2.40 

0.62 

0.6  ±  1.35 

0.5  ±  0.02 

0.85  ±  0.019 

1.1  ±  0.04 

0.81 

1100 

2.39 

0.67 

0.3  ±  1.45 

0.5  ±  0.02 

0.84  ±  0.019 

1.2  ±  0.04 

0.80 

1200 

2.38 

0.75 

-0.2  ±  1.64 

0.6  ±  0.02 

0.83  ±  0.020 

1.3  ±  0.04 

0.82 

1300 

|IKM 

-0.7  ±  1.96 

0.7  ±  0.03 

0.80  ±  0.021 

1.5  ±  0.06 

0.89 

1400 

0.1  ±  2.55 

0.9  ±  0.04 

0.78  ±  0.022 

1.9  ±  0.08 

0.97 
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For  the  M JM llj M j M jl jM j M jl  model  output,  shown  in  Table  98,  the  significant 
effects  of  the  initial  condition  are  negligible  by  the  600th  observation.  For  the  purposes  of 
drawing  inferences.  Table  97  indicates  that  a  truncation  point  about  500  is  best.  The  average 
truncation  point  selected  by  the  MMAE  algorithm  is  488.5. 

The  last  two  of  the  twelve  models,  Time-Sharing  Computer  and  Central  Server  Com¬ 
puter,  are  more  similar  to  applied  discrete-event  simulation  models  than  the  previous  sim¬ 
ulation  of  queuing  systems.  The  output  summaries  for  the  Time-Sharing  Computer  model 
in  Tables  S9  and  100,  and  the  Central  Server  Computer  model  summaries  are  in  Ta¬ 
bles  101  and  102.  Both  of  these  models  have  significant  transient  effects  that  are  relatively 
short-lived,  and  the  MMAE  algorithm  selects  appropriate  truncation  points. 

Overall  in  the  Monte  Carlo  analysis,  the  MMAE  algorithm  selected  truncation  points 
that  achieved  coverage  near  or  better  than  the  best  coverage  achieved  by  a  fixed  truncation 
point.  The  best  results  in  terms  of  coverage  are  achieved  with  the  most  realistic  simulation 
models. 

The  three  lowest  coverage  rates  correspond  to  the  three  models  with  the  highest  av¬ 
erage  estimates  for  ^i.  Since  increased  correlation  in  the  output  sequence  requires  more 
observations  to  achieve  the  same  accuracy  in  the  estimate,  this  seems  to  indicate  that  the 
models  with  high  autocorrelation  at  lag  one  should  have  been  run  longer.  The  next  section 
examines  the  effect  of  increasing  the  run  length. 

6. 4-3  The  Effect  of  Simulation  Run  Length.  A  possible  indication  that  long  simula¬ 
tion  run  lengths  are  necessary  is  an  estimate  for  <t>\  near  one.  Models  with  output  with  high 
autocorrelation  (indicated  by  high  estimates  of  f>i)  require  more  observations  in  each  run  to 
obtain  reliable  estimates.  As  shown  in  Table  103,  the  appears  to  be  inversely  correlated 
with  the  coverage  rate.  Each  of  the  twelve  models  in  the  Monte  Carlo  analysis  are  listed 
in  order  of  decreasing  values  of  the  average  of  the  first  autocorrelation  coefficient  estimate 
(f>i  from  Table  75.  The  next  column  shows  the  resulting  average  coverage  rate  from  the 
truncated  sequences,  as  given  in  Table  78.  In  the  analysis,  the  nominal  coverage  rate  is  0.9. 
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Table  97.  Periodic  Output  Values  Points  for  M JM jl! M j M jlM j M jl  (fXy  —  10.233) 


n 

Avn 

^Vn 

1 

0.00 

0.00 

100 

7.62 

6.24 

200 

8.98 

7.46 

300 

9.60 

8.31 

400 

9.88 

8.90 

500 

10.00 

8.97 

600 

10.26 

9.13 

700 

10.27 

9.30 

800 

10.34 

9.46 

900 

10.46 

9.57 

1000 

10.20 

9.31 

1100 

10.27 

9.50 

1200 

10.26 

9.38 

1300 

10.27 

9.34 

1400 

10.15 

9.32 

Table  98.  Periodic  Truncation  Points  for  MIM f\{M fM flM JM fl  {fXy  =  10.233) 


no 

Av 

PEBIAS  (%  of  ^ly) 

MAD 

Cov 

HW 

0 

9.69 

1.71 

-5.3  ±  0.87 

1.5  ±  0.05 

0.77  ±  0.022 

3.0  ±  0.10 

0.00 

100 

9.98 

1.83 

-2.5  ±  0.93 

1.5  ±  0.06 

0.82  ±  0.020 

3.2  ±  0.11 

0.30 

200 

10.10 

1.93 

-1.3  ±  0.98 

1.5  ±  0.06 

0.82  ±  0.020 

3.3  ±  0.12 

0.58 

300 

10.17 

2.04 

-0.6  ±  1.04 

1.6  ±  0.07 

0.83  ±  0.019 

3.5  ±  0.12 

0.77 

400 

10.21 

2.14 

-0.2  ±  1.09 

1.6  ±  0.07 

0.84  ±  0.019 

3.6  ±  0.13 

0.79 

500 

10.24 

2.25 

0.1  ±  1.14 

1.7  ±  0.08 

0.84  ±  0.019 

3.8  ±  0.14 

0.83 

600 

10.26 

2.36 

0.2  ±  1.20 

1.8  ±  0.08 

0.83  ±  0.019 

3.9  ±  0.14 

0.82 

700 

10.26 

2.47 

0.2  ±  1.26 

1.9  ±  0.08 

0.82  ±  0.020 

4.1  ±  0.15 

0.86 

800 

10.25 

2.60 

0.2  ±  1.32 

2.0  ±  0.09 

0.81  ±  0.020 

4.3  ±  0.16 

0.82 

900 

10.23 

2.72 

0.0  ±  1.38 

2.1  ±  0.09 

0.81  ±  0.021 

4.4  ±  0.17 

0.80 

1000 

10.21 

2.82 

-0.2  ±  1.44 

2.1  ±  0.10 

0.81  ±  0.020 

4.7  ±  0.18 

0.77 

1100 

10.21 

2.92 

-0.2  ±  1.49 

2.2  ±  0.10 

0.82  ±  0.020 

4.9  ±  0.19 

0.75 

1200 

10.16 

3.06 

-0.7  ±  1.56 

2.3  ±  0.10 

0.81  ±  0.021 

5.3  ±  0.20 

0.81 

1300 

10.13 

3.24 

-1.1  ±  1.65 

2.5  ±  0.11 

0.80  ±  0.021 

5.8  ±  0.21 

0.93 
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Table  99.  Periodic  Output  Values  Points  for  Time-Sharing  Computer  Model  —  21.38'' 


n 

Ayn 

1 

0.66 

0.68 

100 

19.21 

17.07 

200 

21.19 

20.49 

300 

21.50 

21.12 

400 

21.67 

20.77 

500 

21.14 

20.84 

600 

21.06 

20.80 

700 

21.15 

19.83 

800 

20.99 

20.34 

900 

21.57 

21.34 

1000 

21.43 

21.06 

1100 

21.69 

21.54 

1200 

21.03 

20.55 

1300 

20.64 

20.27 

1400 

20.96 

20.30 

Table  100.  Periodic  Truncation  Points  for  Time-Sharing  Computer  Model  (/i„  =  21.38) 


Ay 

^^1 

PEBIAS  (%  of  fly) 

MAD 

Cov 

HW 

Bias  Test 

■a 

-3.0  ±  0.14 

0.7  ±  0.02 

0.75  ±  0.022 

1.2  ±  0.02 

0.00 

■Bill 

0.62 

-0.2  ±  0.15 

0.5  ±  0.02 

0.91  ±  0.015 

1.2  ±  0.02 

0.55 

200 

21.38 

0.65 

0.0  ±  0.16 

0.5  ±  0.02 

0.91  ±  0.015 

1.3  ±  0.02 

0.70 

300 

21.38 

0.68 

0.0  ±  0.17 

0.5  ±  0.02 

0.90  ±  0.015 

1.3  ±  0.02 

0.75 

400 

21.38 

0.71 

0.0  ±  0.17 

0.6  ±  0.02 

0.91  ±  0.015 

1.4  ±  0.03 

0.78 

500 

21.38 

0.75 

0.0  ±  0.18 

0.6  ±  0.02 

0.90  ±  0.015 

1.5  ±  0.03 

0.80 

600 

21.39 

0.78 

0.0  ±  0.19 

0.6  ±  0.03 

0.90  ±  0.016 

1.5  ±  0.03 

0.83 

700 

21.39 

0.82 

0.0  ±  0.20 

0.6  ±  0.03 

0.89  ±  0.016 

1.6  ±  0.03 

0.84 

800 

21.39 

0.86 

0.0  ±  0.21 

0.7  ±  0.03 

0.90  ±  0.015 

1.7  ±  0.03 

0.81 

900 

21.38 

0.91 

0.0  ±  0.22 

0.7  ±  0.03 

0.90  ±  0.016 

1.9  ±  0.03 

0.64 

1000 

21.38 

0.98 

0.0  ±  0.24 

0.8  ±  0.03 

0.89  ±  0.016 

2.0  ±  0.04 

0.78 

1100 

21.40 

1.10 

0.1  ±  0.27 

0.9  ±  0.03 

0.91  ±  0.015 

2.2  ±  0.04 

0.99 

1200 

21.36 

1.28 

-0.1  ±  0.31 

1.0  ±  0.04 

0.90  ±  0.016 

2.6  ±  0.05 

1.00 

1300 

21.35 

1.50 

-0.2  ±  0.36 

1.2  ±  0.05 

0.89  ±  0.016 

3.0  ±  0.06 

0.99 
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Table  101.  Periodic  Output  Values  Points  for  Central  Computer  Model  (fiy  =  10.0) 


n 

Avn 

^Vn 

1 

0.97 

0.98 

100 

9.96 

5.30 

200 

10.00 

5.23 

300 

10.12 

5.31 

400 

10.04 

5.32 

500 

10.00 

5.28 

600 

10.06 

5.40 

700 

10.07 

5.38 

800 

10.11 

5.26 

900 

9.92 

5.34 

1000 

10.10 

5.34 

1100 

10.07 

5.45 

1200 

10.03 

5.41 

1300 

9.91 

5.18 

1400 

10.01 

5.25 

Table  102.  Periodic  Truncation  Points  for  Central  Computer  Model  {fiy  =  10.0) 


no 

Ay 

PEBIAS  (%  of  Hy) 

MAD 

Cov 

HW 

Bias  Test 

0 

9.97 

0.11 

-0.3  ±  0.06 

0.1  ±  0.00 

0.87  ±  0.017 

0.2  ±  0.00 

0.00 

100 

10.00 

0.11 

0.0  ±  0.06 

0.1  ±  0.00 

0.90  ±  0.016 

0.2  ±  0.00 

0.45 

200 

10.00 

0.12 

0.0  ±  0.06 

0.1  ±  0.00 

0.90  ±  0.016 

0.2  ±  0.00 

0.62 

300 

10.00 

0.13 

0.0  ±  0.07 

0.1  ±  0.00 

0.89  ±  0.016 

0.2  ±  0.00 

0.77 

400 

10.00 

0.13 

0.0  ±  0.07 

0.1  ±  0.00 

0.89  ±  0.016 

0.3  ±  0.00 

0.85 

500 

10.00 

0.14 

0.0  ±  0.07 

0.1  ±  0.00 

0.89  ±  0.016 

0.3  ±  0.00 

0.90 

600 

10.00 

0.15 

0.0  ±  0.08 

0.1  ±  0.00 

0.90  ±  0.016 

0.3  ±  0.01 

0.95 

700 

10.00 

0.15 

0.0  ±  0.08 

0.1  ±  0.00 

0.90  ±  0.016 

0.3  ±  0.01 

0.96 

800 

10.00 

0.17 

0.0  ±  0.09 

0.1  ±  0.01 

0.88  ±  0.017 

0.3  ±  0.01 

0.96 

900 

10.00 

0.18 

0.0  ±  0.09 

0.1  ±  0.01 

0.89  ±  0.016 

0.3  ±  0.01 

0.94 

1000 

10.00 

0.20 

0.0  ±  0.10 

0.2  ±  0.01 

0.89  ±  0.016 

0.4  ±  0.01 

0.97 

1100 

10.00 

0.21 

0.0  ±  0.11 

0.2  ±  0.01 

0.89  ±  0.016 

0.4  ±  0.01 

1.00 

1200 

10.00 

0.25 

0.0  ±  0.13 

0.2  ±  0.01 

0.88  ±  0.017 

0.5  ±  0.01 

0.99 

1300 

10.00 

0.30 

0.0  ±  0.16 

0.2  ±  0.01 

0.90  ±  0.016 

0.6  ±  0.01 

0.99 
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Table  103.  Autocorrelation  and  Coverage  Relationship 


Model 

EB 

Cov 

M/M/1  p  =  0.95 

0.91 

0.56 

M/M/1  p  =  0.9 

0.91 

0.83 

M/M/l/M/M/l/M/M/1 

0.88 

0.84 

M'M/l  p  =  0.8 

0.88 

0.86 

M/M/1  L, 

0.88 

0.86 

M/M/1  Lq  =  10 

0.88 

0.87 

EJM/1 

0.86 

0.89 

M/M/2 

IBI 

0.88 

MIMIA 

mi 

0.86 

M/M/1  LIFO 

vm 

Central  Server  Computer 

Time-Sharing  Computer 

Correlation  coefficient  equals  -0.38. 


As  the  average  estimated  value  of  increases,  the  realized  coverage  rate  decreases.  In  fact, 
from  the  1,000  Monte  Carlo  sets  of  runs,  the  averages  of  ^i,  ,  and  the  coverage  rates  have 

a  correlation  coefficient  of  -0.38.  Large  positive  autocorrelation,  appcirent  by  close  to 
1.0,  indicate  that  each  additional  observation  provides  less  information.  Therefore,  models 
with  large  positive  autocorrelation  in  their  output  sequences  must  have  more  observations 
to  achieve  the  same  statistical  precision. 

Since  the  model  with  the  highest  average  value  for  and  lowest  coverage  is  MfMfl 
with  p  =  0.95,  the  run  length  for  that  model  is  increased.  Table  104  depicts  the  parameter 
estimates  as  the  run  length  is  increased.  Longer  run  lengths  result  in  less  bias  and  less 
variance  in  the  initial  mean  estimate  y.  The  other  parameter  estimates  do  not  change 
significantly  as  the  run  length  increased. 

The  MMAE  algorithm  selects  truncation  points  for  an  increasing  percentage  of  the 
trials  as  the  run  length  increases.  Table  105  shows  that  the  average  selected  truncation 
point  no  moves  slightly  toward  the  beginning  of  the  sequence  as  the  run  length  increases. 
Table  106  reveals  that  all  of  the  performance  measures,  except  Schruben’s  intial  bias  test, 
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Table  104.  M/M/1  p  =  0.95  Pareimeter  Estimation  Summary  [py  =  18.05) 


Run  Length 

Py 

■ 

m 

m 

■ 

2,500 

|K«1J 

17.45 

5.97 

0.93 

0.05 

0.06 

■nuj 

1.00 

0.00 

5,000 

17.99 

5.34 

0.94 

0.04 

0.05 

1.00 

0.00 

15,000 

18.05 

18.07 

3.71 

0.96 

0.03 

0.03 

1.00 

0.00 

18.05 

17.97 

3.15 

0.96 

0.03 

0.03 

1.00 

0.00 

25,000 

18.05 

2.84 

lifiR 

0.02 

1.00 

0.00 

18.05 

17.95 

2.62 

0.02 

1.00 

0.00 

35,000 

18.05 

2.36 

0.02 

1.00 

0.00 

Table  105.  Truncation  Points  tiq 


Run  Length 

Avg 

St  Dev 

2,500 

24 

2391 

682.6 

493.4 

5,000 

35 

3673 

783.9 

589.1 

15,000 

58 

4932 

707.2 

559.0 

61 

648.1 

489.0 

25,000 

58 

5463 

634.8 

486.5 

38 

3758 

607.6 

426.7 

35,000 

5341 

590.3 

427.7 
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Table  106.  Truncation  Point  hp  Evaluations 


PEBIAS  (%  of  fjLy) 

MAD 

Cov 

Half  Width 

Bias  Test 

-15.0 

± 

1.32 

4.0 

± 

0.16 

0.67 

± 

0.029 

6.9 

± 

0.27 

5,000 

-6.6 

± 

1.13 

3.1 

0.13 

0.76 

± 

0.024 

6.5 

± 

0.23 

■H 

15,000 

-1.1 

± 

0.79 

2.1 

± 

0.09 

0.84 

± 

0.019 

4.8 

± 

0.15 

0.05 

-0.9 

± 

0.69 

1.9 

± 

0.08 

0.84 

± 

0.019 

4.2 

± 

0.13 

0.03 

25,000 

-0.7 

± 

0.62 

1.7 

± 

0.07 

0.85 

± 

0.018 

3.9 

± 

0.11 

0.02 

-0.7 

± 

0.55 

1.5 

0.06 

0.86 

± 

0.018 

3.5 

± 

0.09 

0.01 

35,000 

-0.6 

± 

0.50 

1.4 

± 

0.05 

0.86 

± 

0.018 

3.2 

± 

0.08 

0.01 

improved  with  increased  run  lengths.  Perhaps  the  estimate  of  (f>i  may  provide  a  basis  for  a 
“rule  of  thumb”  to  determine  a  reasonable  simulation  run  length. 

6.4-4  'The  Effect  of  Filter  Spacing.  Between  using  the  estimate  k  instead  ol  k  =  1.0 
and  increasing  the  run  length,  the  MMAE  algorithm  has  a  very  low  rate  of  failure  to  select 
a  truncation  point  for  all  of  the  models  except  the  M/M/4  queue.  Tables  95  and  96  indicate 
that  the  M/M/4  transient  is  effectively  complete  by  the  500th  observation.  The  average  value 
for  is  0.64,  relatively  low,  so  the  run  length  of  1,500  observations  is  sufficient.  Compared 
to  the  other  simulation  models,  the  M/M/4  output  has  less  variance.  Less  variance  results 
in  the  minimum,  mean,  and  meiximum  output  values  being  closer.  Since  these  values  are 
used  to  space  the  MMAE  filters,  tighter  filter  spacing  results.  If  the  filters  are  spaced  too 
close  together,  the  filter  probabilities  fluctuate  between  filters. 

The  lower  filter  is  positioned  at  consecutively  lower  fixed  values  to  test  the  effect  of 
increasing  the  filter  spacing.  Only  the  lower  filter  is  moved  since  the  transient  bias  induced 
by  the  empty  and  idle  initial  conditions  result  in  low  output  values.  Table  107  shows  that, 
as  the  lower  filter  is  moved  downward,  the  MMAE  algorithm  selects  a  truncation  point  on 
more  trials.  The  average  truncation  point  is  earlier  in  the  sequence  as  the  filter  spacing 
increases.  Table  108  depicts  the  effects  of  these  earlier  truncation  points  on  the  performance 
measures.  The  PEBIAS  only  has  the  same  sign  as  the  transient  values  when  the  low  filter 
is  at  -4.0.  The  MAD  improves  with  each  increase  in  spacing.  The  coverage  appears  to  be 
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Table  107.  M/M/4  Truncation  Points  Uo 


Low  Filter  Position 

Fail 

Min 

Max 

Avg 

St  Dev 

y  -  0.9(j/  -  2/rnin) 

mm 

14 

1493 

513.6 

408.0 

0 

18 

1493 

524.9 

409.0 

-1 

fm 

1 

1491 

499.2 

386.6 

-2 

0.109 

1 

1485 

367.1 

391.5 

-3 

0.001 

1 

1474 

113.7 

274.0 

-4 

0.000 

1 

1058 

11.5 

88.6 

Table  108.  MfMfA  Truncation  Point  ho  Evaluations 


Low  Filter  Position 

PEBIAS  (%  of  fly) 

MAD 

Cov 

Half  Width 

Bias  Test 

y  -  0.9(j/  -  j/jnin) 

8.3 

± 

2.40 

0.5 

± 

0.05 

0.86 

± 

0.023 

1.1 

± 

0.08 

0.76 

0 

8.5 

± 

2.41 

0.5 

i: 

0.05 

0.87 

± 

0.022 

1.1 

± 

0.08 

0.76 

-1 

7.8 

± 

2.07 

0.5 

± 

0.04 

0.86 

± 

0.021 

1.1 

± 

0.07 

0.77 

-2 

7.0 

± 

1.72 

0.5 

± 

0.03 

0.85 

± 

0.020 

1.0 

± 

0.06 

0.65 

-3 

2.7 

1.36 

0.4 

± 

0.03 

0.84 

± 

0.019 

0.9 

± 

0.04 

0.17 

-4 

-1.0 

± 

0.96 

0.3 

± 

0.02 

0.85 

± 

0.019 

0.7 

± 

0.03 

0.02 
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decreasing,  but  not  a  statistically  significant  amount.  The  half  widths  are  smaller  because 
of  the  increcise  in  the  number  of  untruncated  observations. 

For  the  case  of  M/M/4,  the  increased  filter  spacing  reduces  the  MMAE  algorithm’s 
rate  of  failure  to  select  a  truncation  point.  However,  extreme  filter  spacing,  which  results 
in  selecting  trucation  points  very  early  in  the  sequence,  may  have  an  adverse  effect  on  the 
coverage  rate  for  the  truncated  sequences. 

6. 5  Summary 

This  chapter  proposes  an  algorithm  to  select  the  truncation  point  to  eliminate  the 
bias  induced  by  initial  conditions  for  multiple  replications  of  discrete-event  simulations.  The 
procedure  is  based  on  MMAE  with  the  necessary  parameters  estimated  using  the  last  half  of 
the  output  sequences.  After  estimating  an  assumed  steady-state  AR(2)  with  measurement 
noise  model,  three  Kalman  filters  are  initialized.  The  Kalman  filters  are  spaced  based  on 
the  minimum,  mean,  and  maximum  value  of  the  averaged  output  values.  By  processing  the 
average  output  values  through  the  bank  of  Kalman  filters,  the  truncation  point  is  selected 
when  the  time- varying  MMAE  mean  estimate  is  within  a  small  tolerance  of  the  mean  estimate 
for  five  consecutive  observations.  The  algorithm  performs  well  with  the  steady-state  model 
estimated  with  the  last  half  of  the  sequences  and  the  small  tolerance  on  the  mean  set  to  1.0. 

This  approach  is  tested  on  twelve  models  in  a  Monte  Carlo  analysis  of  1,000  sets  with 
each  set  using  5  replications.  The  selected  truncation  points  result  in  truncated  sequences 
that  provided  excellent  estimates  of  the  steady-state  values.  The  method  appears  to  work 
for  both  types  of  output  sequences:  “statistics  based  on  observations”  and  “statistics  based 
on  time-persistent  variables”.  In  the  first  case,  the  state  transition  matrix  used  in  the 
Kalman  filter  is  the  relationship  between  sequenced  observations.  For  the  periodic  samples 
of  time-persisent  variables,  the  state  transition  matrix  is  the  relationship  between  samples 
equally  spaced  in  time.  In  both  cases,  the  parameters  are  estimated  in  the  same  manner. 
The  MMAE-selected  truncation  points  perform  slightly  better  than  Kelton  and  Law’s  [44] 
truncation  points. 
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This  approach  has  several  advantages  over  Kelton  and  Law’s  technique.  First,  the 
analyst  has  fewer  parameters  to  specify.  The  decisions  to  make  in  applying  the  MMAE 
algorithm  concern  the  filter  spacing  and  the  tolerances  on  the  MMAE  mean  estmate  to 
select  the  truncation  point.  The  tested  values  for  the  MMAE  algorithm,  which  are  selected 
without  attempting  to  optimize,  appear  to  provide  relatively  robust  results.  In  addition, 
these  algorithm  settings  can  easily  be  determined  from  the  output  sequences  for  the  analyst. 
Second,  as  seen  in  Chapter  V  with  the  time  series  models,  this  approach  works  for  non¬ 
monotonic  convergence  to  the  steady-state  mean.  Third,  rather  than  having  to  restart 
the  simulation  models,  such  as  Kelton  and  Law’s  technique,  the  analyst  can  specify  the 
replication  run  length.  However,  if  a  sequential  approach  that  continues  the  replication  runs 
is  desired,  the  MMAE  approach  can  be  modified  easily. 
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VII.  Multivariate  Steady-State  Identification 

7. 1  Introduction 

This  chapter  extends  the  univariate  initial  data  truncation  methodology  from  Chap¬ 
ter  VI  to  application  for  multivariate  output  sequences.  A  first-order  vector  autoregressive 
with  measurement  noise  model  is  proposed  to  approximate  the  simulation  output  sequences. 
An  estimation  scheme  based  on  the  sample  autocorrelation  matrices  is  presented.  Using 
the  estimated  model  and  parameters,  Multiple  Model  Adaptive  Estimation  (MMAE)  with  a 
bank  of  Kalman  filters  is  applied.  The  truncation  point  is  selected  when  a  vector  norm  of  a 
mean  estimate  minus  the  MMAE  time- varying  mean  estimate  are  within  a  small  tolerance. 
The  multivariate  algorithm  is  summarized  and  tested. 

7.2  VAR(l)  with  Measurement  Noise  Model 

For  univariate  output  sequences,  dteudel  and  Wu  [97]  and  Schriber  and  Andrews  [85] 
report  that  periodic  samples  of  M/M/1  queue  length  can  be  adequately  represented  with  an 
AR(1)  model.  Charnes  [12]  finds  that  multivariate  output  sequences  of  queue  lengths  can  be 
represented  as  a  first-order  vector  autoregressive,  VAR(l),  model.  In  this  application,  the 
proposed  model  assumes  the  filter-design  states  have  a  VAR(l)  formulation,  but  the  states 
are  observed  with  an  independent  additive  white  noise  vector.  Thus,  multivariate  simulation 
output  is  formulated  as  a  VAR(l)  with  measurement  noise  model.  The  proposed  model  is 
linear  with  time-invariant  system  matric  -;s  and  constant  noise  covariances. 

The  average  of  multivariate  sequences  with  S  responses  is  modeled  as  follows.  .  et 
x{tn)  represent  the  5x1  state  vector,  ^  be  the  5x5  time-invariant  state  transition  matrix, 
and  wj(tn)  be  the  5x1  white  noise  vector,  which  is  jointly  normally-distributed  with  a  zero 
mean  vector  and  an  5  x  5  covariance  matrix  of  Qj.  The  first-order  vector  autoregressive 
model  VAR(l)  for  the  filter-design  state  vector  is 

x(t„)  =  ^x(t„-i)  -h  Wj(t„) 
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In  this  application,  the  state  vector  is  time- varying  representations  of  each  of  the  S  responses 
corrected  for  that  response’s  mean.  For  example,  if  the  first  simulation  response  is  queue 
length,  the  first  element  of  the  state  vector  Xi(t„)  is  the  modeled  queue  length  at  time 
minus  the  expected  queue  length. 

This  formulation  is  a  VAR(l)  with  measurement  noise  model,  rather  than  a  pure 
VAR(l),  because  the  states  are  observed  only  with  the  addition  of  a  vector  of  measurement 
noise.  Define  the  5x1  measurement  vector  z(t„)  as  the  filter-design  state  vector  x(t„)  plus 
an  5  X  1  measurement  noise  vector  v(t„).  Thus,  the  measurement  matrix  H  is  simply  an 
5x5  identity  matrix.  The  associated  measurement  model  is 

z(tn)  =  Hx(t„)  -I-  v(t„) 

=  x(t„)  -f  v(t„) 

The  simulation  sequence  averaged  over  the  replications  {yn}  is  used  to  determine  realizations 
of  the  measurements: 

Zn  =  yn  -  Ay  (68) 

While  significant  lag  effects  in  many  simulation  models  may  occur  at  greater  than  one 
lag,  Charnes  [12]  reports  that,  for  periodic  samples  of  queue  lengths,  a  VAR(l)  formulation 
is  sufficient  to  characterize  the  autocorrelation  structure.  The  increased  flexibility  of  the 
VAR(l)  with  measurement  noise  model  should  be  sufficient  for  similar  types  of  output  se¬ 
quences.  However,  analysts  should  carefully  consider  the  applicability  of  this  methodology 
to  untested  types  of  simulation  models.  Specifically,  if  each  of  the  response  sequences  does 
not  exhibit  significant  autocovariances  at  one  and  two  lags,  the  inverse  matrices  necessary 
for  estimation  may  not  exist. 

7.S  Parameter  Estimation 

In  similar  fashion  as  the  univariate  case,  parameter  estimates  are  based  on  the  sample 
autocorrelations.  Assuming  covariance  stationarity,  the  autocovariance  matrices  at  a  lag  of  i 
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for  the  sequence  of  5  x  1  averaged  output  vectors  {y(ti),  y(t2), y(tjv)}  are  defined  as 


ryy(i)  =  E[(y(t„)  -  /iy)  (y(t„..)  -  ] 


The  theoretical  covariance  matrix  and  first  two  lagged  autocovariance  matrices  of  y(t„) 
are  derived  below.  Because  the  dynamics  noise  sequence  {wi(t„)}  and  the  measurement  noise 
sequence  {v(t„)}  are  each  modeled  as  zero  mean,  independent  in  time,  and  independent  of 
each  other  and  the  state  vectors,  many  of  the  cross  terms  have  zero  expected  value: 

Syy  =  ryy(O) 

=  E[(y(t„)-/Xy)(y(t„)-;.yf] 

=  E[(x(t„)  +  v(tO)(x(t„)  +  v(f„)f] 

=  E[x(t„)x^(t„)]  +  E[v(t„)x^(t„)]  +  E[v(t„)x^(t„)]  +  E[v(t„)v^(t„)] 

=  E[x(t„)x^(t„)]  +  E[v(t„)v^(t„)] 

=  rxx(O)  +  R 

=  P-  +R 


Since  the  measurement  matrix  H  equals  the  identity  matrix,  the  covariance  of  output  se¬ 
quence  is  equal  to  the  Kalman  filter  predicted  residual  variance,  HP~H^  R.  The  autoco- 
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variance  matrix  at  one  lag  ryy(l)  is 

ryy(l)  =  E[(y(^n)  -  fiy)  (y(^n-l)  -  MyD 

=  E[(x(t„)  +  \{tn))  (x(tn-l)  +  v{tn-l)f] 

=  E[x(tn)x^(t„-l)]  +  E[v(i„)x^(t„_l)]  +  E{v(i„)x^(t„_l)]  +  E[v(in)v^(t„_l)] 

=  E[x(tn)x^(tn-l)] 

=  rxx(i) 

=  E[($x(t„-i)  +  Wd(t„))x’’(tn-l)] 

=  $E[x(i„_i)x^(t„-i)] 

=  ^rxx(O) 

=  $P- 

Similarly,  the  autocovariance  matrix  at  lag  two  are  derived: 

ryy(2)  =  E[(y(t„)  -  fly)  (y(tn-2)  -  My)’’] 

=  E[(x(tn)  +  v(t„))(x(t„_2)  +  v(t„_2)f  ] 

=  E[x(tn)x^(t„_2)] 

=  rxx(2) 

=  E[($x(i„_i)  +  Wrf(<„))x^(t„_2)] 

=  $E[x(t„_i)x^(t„-2)] 

=  ^rxx(i) 

=  ^'rxx(o) 

=  $2p- 

Using  these  relationships,  the  state  transition  matrix  $  and  the  Kalman  filter  gain 
matrix  K  are  determined  in  terms  of  the  autocovariance  matrices: 

ryy(2)r-i(i)  =$^p-($p-)-^ 

=  (69) 

=  $ 
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From  Equation  (30)  on  page  58,  the  Keilman  filter  gain  K  are  derived  as  follows: 

K  =  P-H^(HP-H’'  +  RJ-' 

=  p-|p-  +  Rr' 

=  $-‘r„(i)r-j(o)  (70) 

=  (r„(2)rp}(i)i-‘  r„(i)r-j(0) 

=  r„(i)r;;(2)ryy(i)r->(0) 

The  inverse  matrices  necessary  for  estimation  (ryy(i)  for  2  =  0, 1,2)  very  likely  exist  if  each 
of  the  output  sequences  has  significzmt  values  for  its  first  and  second  autocovariances.  If 
each  of  these  sequence  exhibits  AR(1)  behavior,  the  first  two  autocovariances  probably  are 
not  zero. 

Using  Equations  (69)  and  (70)  and  the  estimated  autocovariance  matrices  based  on 
the  last  half  of  the  output  sequence, 

ryy(i)=y^^  E  =  (7i) 

n=^+l 

From  these  sample  autocovariance  matrices,  the  state  transition  matrix  and  the  Kalman  filter 
gain  vector  can  be  estimated.  These  are  the  matrices  necessary  to  apply  the  Kalman  filter. 
Applying  the  Kalman  filter  to  the  l^lst  half  of  the  sequence  can  determine  the  filter-design 
residual  covariance  matrix: 

HP-H’'  +  R=^  £  r„rj  (72) 

n=^  +  l 

Using  the  last  half  of  the  output  sequence,  an  assumed  steady-state  mean  y  is  calculated 

and  the  minimum  and  maximum  of  each  response  are  found.  All  the  estimates  necessary  to 

T 

apply  MMAE  ($,  K,  HP-H'  -f  R,  and  y)  are  available.  After  spacing  and  initializing  the 
bank  of  filters,  MMAE  can  be  applied.  The  L  =  3^  filters  are  positioned  at  each  combination 
of  the  minimum,  mean,  and  maximum  of  the  S  responses.  The  initial  filter  state  vectors 
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are  zeros.  As  in  the  univariate  case,  the  MMAE  time-varying  mean  estimate  is 

compared  with  y  with  a  vector  norm  to  determine  the  appropriate  truncation  point.  When 
the  Euclidean  distance  of  ||/iy(tn)  —  y||  is  less  than  small  tolerance  for  a  few  consecutive 
observations,  the  truncation  point  is  selected.  The  small  tolerance  should  probably  be  a 
small  percent,  perhaps  ten,  of  the  vector  norm  of  the  sample  mean,  but  in  these  tests,  an 
absolute  value  of  one  is  used.  Achieving  the  criterion  for  several  consecutive  observations 
reduces  the  chance  that  the  randomness  of  the  output  sequence  might  give  a  false  indication 
that  its  transient  is  eflFectively  complete.  The  next  section  summarizes  the  specific  steps  in 
the  technique. 

7.4  The  Multivariate  Truncation- Point  Selection  Algorithm 

The  truncation-point  selection  algorithm  for  multivariate  output  sequences  is  summa¬ 
rized  below. 

Step  1.  Select  the  simulation  run  length  N  and  number  of  replications  M.  Run  M  repli¬ 
cations  of  the  simulation  of  length  N,  storing  the  multivariate  sequences  each  with  S 
responses. 

Step  2.  Construct  a  sequence  of  length  N  of  the  average  across  the  M  replications  for  each 
of  the  5  responses. 

Step  3.  Based  on  the  last  half  of  the  averaged  sequence,  determine  an  estimate  of  the 
steady-state  mean  vector  y,  the  covariance  matrix  and  autocovariance  matrices  with 
Equation  (71).  Based  on  Equations  (69)  and  (70),  estimate  4  and  K.  Estimate 
HP“H^  -f  R  with  Equation  (72). 

Step  4.  Find  the  minimum  and  maximum  of  each  of  the  S  responses  in  the  last  half  of  the 
sequence  of  averages.  Initialize  an  MMAE  bank  of  L  =  3^  filters  with  a  filter  at  each 
combination  of  the  minimums,  means,  and  maximums  of  the  S  responses.  The  initial 
state  estimates  x(to)  are  zero  vectors,  and  the  initial  MMAE  filter  probabilities  pi{to) 
are  i. 
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Step  5,  Process  the  observations  in  the  averaged  sequence  through  the  MMAE  bank  of 
filters.  For  each  observation,  propagate  the  estimated  state  vector  in  each  filter  us¬ 
ing  Equations  (22)  and  (45).  Update  the  state  estimates  with  Equation  (25),  using 
the  observation  from  the  averaged  sequence,  y„,  minus  that  filter’s  mean  /iy  as  the 
measurement  z„.  Calculate  the  MMAE  filter  probabilities  Pj(tn)  with  Equation  (36). 
Using  the  MMAE  filter  probabilities,  calculate  the  weighted  MMAE  mean  estimate 
fiy{tn)  by  Equation  (39)  where  a(t„)  =  /ty(t„).  Reset  any  near  zero  MMAE  filter 
probabilities  to  a  very  small  positive  value,  such  as  When  ||/Xy(t„)  —  y||  <  1.0 

for  5  consecutive  observations,  select  the  truncation  point  no  as  (In  these  tests, 
the  Euclidean  norm  is  used,  the  small  tolerance  is  1.0  and  5  consecutive  observations 
are  used.  A  relative  criterion,  such  as  the  absolute  difference  being  less  than  a  small 
percentage  of  the  vector  norm  of  the  sample  mean,  is  probably  more  appropriate  in 
general  applications.) 

7.5  Monte  Carlo  Results 

A  Monte  Carlo  analysis  is  conducted  using  the  MMAE  and  Schruben’s  truncation-point 
selection  algorithms.  Four  simulation  models  are  used  to  generate  the  output  sequences.  For 
each  of  1,000  sets,  10  replications  of  1,500  observations  each  are  generated. 

The  simulation  models  used  to  test  this  algorithm  are  simulations  of  open  queuing 
systems  and  closed  queuing  systems.  An  open  queuing  system  has  new  entities  arriving, 
being  processed,  and  departing.  The  multivariate  output  responses  are  a  periodic  sampling 
of  the  queue  lengths  at  each  server.  If  the  queues  are  M/M/1  queues  in  tandem,  Ross 
[79:326-330]  shows  that  the  queue  lengths  are  independent  and  that  the  expected  queue 
lengths  for  each  server  are  the  same  as  for  stand-alone  M/M/1  queues. 

In  contrast  to  the  open  queuing  system,  a  closed  queuing  system  has  a  fixed  number  of 
entities  cycling  through  a  circle  of  servers.  The  multivariate  output  responses  are  a  periodic 
sampling  of  the  number  of  customers  at  each  server  for  all  but  one  of  the  servers.  If  each 
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queue  has  exponentially  distributed  service  times,  Ross  [79:330-335]  provides  the  equations 
to  calculate  their  analytic  steady-state  means. 

The  third  type  of  multivariate  output  is  from  the  open  queuing  model.  Using  two 
queues,  the  multivariate  output  consists  of  the  the  waiting  time  plus  service  time  at  the 
first  server  and  the  total  system  time.  In  contrast  to  the  two  previous  cases  which  are 
“statistics  based  on  time-persistent  variables”,  these  output  sequence^  are  “statistics  based 
on  observations”. 

The  evaluation  measures  are  based  on  the  multivariate  method  of  replications.  On  the 
truncated  sequences,  the  confidence  region  is  calculated  with  Equation  (4)  and  associated 
volume  with  Equation  (5).  Over  1,000  sets  of  runs,  the  coverage  rate  for  the  analytic  mean 
and  average  confidence  region  volumes  are  reported. 

For  comparison,  Schruben’s  multivariate  truncation  algorithm,  shown  in  Equations  (14), 
(15),  ana  ^  '),  also  is  applied  with  a  =  0.1.  For  the  sequences  truncated  with  Schruben’s  al¬ 
gorithm,  the  confidence  regions  are  calculated.  The  associated  coverage  rates  and  confidence 
region  volumes  are  summarized. 

The  first  simulation  model  is  an  open  model  of  three  M/M/1  queues  in  tandem.  The 
interarrival  time  to  the  first  queue  in  the  series  is  exponentially  distributed  with  a  mean 
of  1.0.  The  queues  have  exponentially  distributed  service  times  of  0.5,  0.7,  and  0.9  time 
units,  respectively.  Ross  [79:326-330]  shows  that  the  length  of  queue  for  each  server  is  the 
same  as  for  an  M/M/1  queue  and  that  the  queue  lengths  are  independent  of  each  other. 
The  responses  are  the  queue  lengths  with  analytic  steady-state  solutions  of  (0.5,  1.63,  8.1)^. 
Two  sets  of  initial  conditions  are  tested.  The  first  is  all  empty  queues  and  idle  servers.  The 
second  set  of  initial  conditions  for  this  model  is  that  of  idle  servers  but  with  26  customers  in 
the  first  queue.  In  the  first  test,  the  system  builds  to  steady  state.  In  contrast,  in  the  second 
test,  the  system  starts  overcongested,  and  the  output  sequences  decays  to  their  steady-state 
levels.  The  output  responses  are  the  queue  lengths  measured  at  every  time  interval  for  1,500 
observations. 
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Table  109.  Open  Model  Queue  Lengths  Parameter  Estimation  Summary 


Model 

Open  Model  (Empty  and  Idle) 

0.50  0.50  0.03 

1.63  1.64  0.15 

8.10  8.09  1.71 

Open  Model  (Lg^(to)  =  26) 

0.50  0.50  0.03 

1.63  1.64  0.15 

8.10  8.12  1.74 

Figure  21.  Average  Estimated  Covariance  Matrix  Eyy  ) 

0.124  (0.021)  -0.003(0.023)  -0.018(0.082)' 

-0.003  (0.023)  0.638  (0.170)  -0.088  (0.334) 

-0.018  (0.082)  -0.088  (0.334)  5.874  (3.333) 

The  steady-state  model  is  estimated  with  the  last  half  of  the  sequence.  Over  the  1,000 
Monte  Carlo  sets  of  runs,  Table  109  shows  that  the  estimated  steady-state  means  are  very 
near  the  analytic  solutions.  Figure  21  shows  the  average  covariance  matrix  and  standard 
deviation  (not  the  standard  error)  for  the  1,000  sets  with  empty  and  idle  initial  conditions. 
The  diagonal  elements  indicate  that  the  queue  length  variance  increases  as  traffic  intensity 
increases.  The  off-diagonal  elements  are  near  zero  because  of  the  independence  between 
queue  lengths.  Figures  22  and  23  show  the  average  autocovariance  matrices  at  one  and  two 
lags,  respectively.  The  queue  lengths  exhibit  positive  autocovariances  and  insignificant 
cross-covariances. 

Figures  24  and  25  show  the  average  state  transition  matrices  $  and  the  average  Kalman 


Figure  22.  Average  Estimated  Autocovariance  Matrix  at  One  Lag  ryy(l)  (dj. 


Vr.Vc 


(1) 


0.087  (0.020) 
0.013(0.023) 
-0.015(0.082) 


0.013  (0.023) 
0.577  (0.169) 
-0.058  (0.333) 


-0.015  (0.082)  ■ 
-0.058  (0.333) 
5.788  (3.334) 
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Figure  23.  Average  Estimated  Autocovariance  Matrix  at  Two  Lags  fyy(2)  (dp^  ^  ^2)  ) 

0.064  (0.019)  0.021  (0.022)  -0.011  (0.081)' 

0.021  (0.022)  0.525(0.167)  -0.033(0.332) 

-0.011  (0.081)  -0.033(0.332)  5.705  (3.334) 


Figure  24.  Average  State  Transition  Matrix  $  (d’#,^  ) 

■  0.713  (0.052)  0.023  (0.012)  0.000  (0.004)  ' 

0.023  (0.012)  0.899  (0.025)  0.005  (0.006) 

0.000  (0.004)  0.005  (0.006)  0.982  (0.010) 

filter  gains  K  for  the  1,000  Monte  Carlo  sets  of  runs.  Because  of  the  small  cross-correlations, 
the  off-diagonal  terms  are  near  zero.  The  diagonal  elements  of  the  state  transition  matrix 
show  that  the  weights  for  propagating  the  estimates  increase  with  the  traffic  intensity  of 
the  queues.  Similarly,  the  Kalman  filter  gains  increase  with  traffic  intensity.  The  diagonal 
elements  of  the  Kalman  filter  are  relatively  near  their  upper  bound  of  1.0.  Since  there  are 
only  insignificant  differences  between  the  average  autocovariance  matrices  for  the  two  initial 
conditions,  only  the  average  matrices  are  shown  for  the  empty  and  idle  initial  conditions. 

Table  110  shows  statistics  on  the  truncation  points  selected  by  the  MMAE  approach 
and  by  Schruben’s  approach.  For  these  tests,  MMAE  selected  a  point  in  1,999  out  of  2,000 
sets,  whereas  Schruben’s  algorithm  failed  to  select  a  truncation  point  for  6  percent  of  the 
sets.  The  MMAE  selected  truncation  points  are  considerably  earlier  in  the  sequences  and 


Figure  25.  Average  Kalman  Filter  Gain  K  (^k,,.  ) 


0.694(0.042)  0.024  (0.011) 
0.024  (0.011)  0.897  (0.023) 
-0.001  (0.004)  0.005  (0.006) 


-0.001  (0.004)  ■ 
0.005  (0.006) 
0.981  (0.010) 
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Table  110.  Truncation  Points  rio 


Fail 

Min 

Max 

Avg 

St  Dev 

Open  Model(Empty  and  Idle) 

MMAE  Algorithm 

24 

1064 

173.5 

147.6 

Schruben  Algorithm 

20 

1480 

353.5 

355.1 

Open 

Model 

[LM 

=  26) 

MMAE  Algorithm 

0.000 

59 

928 

276.9 

158.7 

Schruben  Algorithm 

0.064 

51 

1481 

447.2 

351.0 

Table  111.  Truncation  Point  np  Evaluations 


no  Cov  Vol 

Open  Model  (Empty  and  Idle) 

MMAE  Algorithm 
Schruben  Algorithm 

173.5  ±  7.68 

353.5  ±  19.05 

0.83  ±  0.020  0.44  ±  0.026 
0.82  ±  0.021  1.76  ±  0.614 

0 

pen  Model  (I<,i(to)  =  26) 

MMAE  Algorithm 
Schruben  Algorithm 

276.9  ±  8.25 
447.2  ±  18.87 

0.85  ±  0.019  0.47  ±  0.019 
0.82  ±  0.020  1.58  ±  0.485 

have  less  variance  than  Schruben’s  technique.  Table  111  shows  that  the  MMAE-selected 
truncation  points  result  in  sequences  that  achieve  a  higher  coverage  rate  at  a  smaller  average 
confidence  region  volume. 

Table  112  depicts  the  average  periodic  output  values  when  the  initial  conditions  are 
empty  and  idle.  The  output  increases  to  the  steady-state  values.  The  double  horizontal 
line  indicates  where  the  average  MMAE  selected  truncation  point  occurs,  and  the  single 
horizontal  line  is  where  the  average  truncation  point  selected  by  Schruben’s  algorithm  occurs. 
Table  113  shows  periodic  output  values  for  the  multivariate  sequence  when  the  initial  queue 
length  at  the  first  server  is  26.  Since  the  first  queue  is  initially  over  congested,  each  of  the 
three  queues  becomes  overcongested  during  the  transient.  The  truncated  sequences  resulting 
from  both  algorithms  retain  some  transient  data,  particularly  with  the  initial  queue  of  26 
entities  at  the  first  server. 
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Table  112.  Periodic  Output  Values  Points  for  Open  Model  (Empty  and  Idle) 


n 

^Vn.7 

^Vn.S 

■51 

0.50 

1.63 

8.10 

1 

0.29 

0.68 

0.16 

0.43 

0.03 

0.18 

100 

0.48 

1.09 

1.60 

2.54 

5.53 

5.93 

■a 

0.50 

1.13 

1.65 

2.55 

6.77 

7.38 

■a 

0.50 

1.12 

1.60 

2.57 

7.37 

8.32 

mm 

0.50 

1.12 

1.60 

2.50 

7.57 

8.71 

Bn| 

0.49 

1.08 

1.63 

2.55 

7.77 

8.96 

600 

0.51 

1.12 

1.63 

2.59 

7.99 

9.18 

700 

0.50 

1.12 

1.60 

2.54 

8.12 

9.34 

800 

0.48 

1.09 

1.61 

2.55 

8.06 

9.33 

900 

0.48 

1.10 

1.64 

2.55 

7.98 

9.29 

1000 

0.47 

1.08 

1.65 

2.55 

7.95 

9.24 

1100 

0.52 

1.15 

1.63 

2.59 

7.95 

9.24 

1200 

0.51 

1.14 

1.63 

2.59 

8.10 

9.35 

1300 

0.48 

1.07 

1.62 

2.57 

8.26 

9.49 

1400 

0.50 

1.12 

1.63 

2.52 

8.13 

9.53 

1500 

0.50 

1.13 

1.65 

2.60 

8.26 

9.62 

Table  113.  Periodic  Output  Values  Points  for  Open  Model  (Z<,i(io)  =  26) 


n 

I3IH 

^yn.a 

EH9 

^Vn.S 

f^y 

0.50 

1.63 

8.10 

1 

24.02 

1.72 

0.64 

1.03 

0.11 

0.36 

■a 

0.49 

1.08 

2.29 

3.89 

15.65 

11.22 

ESI 

0.51 

1.11 

1.69 

2.60 

11.79 

12.29 

Ba 

0.48 

1.09 

1.63 

2.58 

9.91 

11.52 

■a 

0.48 

1.06 

1.62 

2.52 

9.32 

10.94 

Ba 

0.49 

1.07 

1.59 

2.51 

8.76 

10.43 

B9 

0.51 

1.14 

1.63 

2.61 

8.61 

10.22 

■a 

0.49 

1.12 

1.60 

2.53 

8.53 

9.94 

800 

0.49 

1.10 

1.65 

2.61 

8.31 

9.76 

900 

0.51 

1.12 

1.60 

2.56 

8.17 

9.47 

1000 

0.50 

1.11 

1.66 

2.57 

7.97 

9.33 

1100 

0.51 

1.11 

1.61 

2.55 

7.99 

9.29 

1200 

0.50 

1.13 

1.62 

2.53 

8.13 

9.35 

1300 

0.51 

1.13 

1.58 

2.49 

8.12 

9.41 

1400 

0.48 

1.10 

1.63 

2.53 

8.14 

9.30 

1500 

0.51 

1.13 

1.63 

2.56 

8.14 

9.36 

247 


Table  114.  Periodic  Truncation  Points 


ho 


0.50  1.63  8.10 


for  Open  Model  (Empty  and  Idle) 


Cov  Vol 


0 

0.50 

1.62 

7.52 

0.79  ±  0.02 

0.30  ±  0.01 

100 

0.50 

1.63 

7.80 

0.82  ±  0.02 

0.35  ±  0.01 

0.50 

1.63 

7.92 

0.83  ±  0.02 

0.40  ±  0.02 

Bi 

0.50 

1.63 

7.99 

0.84  ±  0.02 

0.45  ±  0.02 

400 

0.50 

1.63 

8.03 

0.83  ±  0.02 

0.51  ±  0.02 

500 

0.50 

1.64 

8.07 

0.83  ±  0.02 

0.58  ±  0.02 

600 

0.50 

1.64 

8.09 

0.85  ±  0.02 

0.67  ±  0.03 

700 

0.50 

1.64 

8.09 

0.85  ±  0.02 

0.79  ±  0.03 

800 

0.50 

1.64 

8.09 

0.83  ±  0.02 

0.94  ±  0.04 

900 

0.50 

1.64 

8.10 

0.83  ±  0.02 

1.15  ±  0.05 

1000 

0-50 

1.64 

8.14 

0.82  ±  0.02 

1.43  ±  0.06 

1100 

0.50 

1.64 

8.19 

0.81  ±  0.02 

1.88  ±  0.08 

1200 

0.50 

1.64 

8.22 

0.80  ±  0.02 

2.65  ±  0.11 

1300 

0.50 

1.64 

8.24 

0.77  ±  0.02 

4.27  ±  0.19 

1400 

0.50 

1.64 

8.23 

0.77  ±  0.02 

8.79  ±  0.45 

Table  114  shows  for  the  periodic  truncated  sequences,  the  average  estimated  responses 
along  with  the  corresponding  coverage  rate  and  confidence  region  volume  for  the  open  model 
with  emwty  ajid  idle  initial  conditions.  Table  115  shows  the  same  statistics  when  the  initial 
conditioi'.s  are  26  customers  at  the  first  server.  The  estimated  responses  converge  to  the 
analytic  values  by  the  800th  observation.  However,  the  mean  estimator  error  is  very  small 
consider  bly  earlier  in  the  sequences.  The  coverage  rate  begins  at  zero  with  no  truncation, 
but  is  very  near  the  nominal  rate  of  0.9  after  truncating  only  100  observations.  The  coverage 
rate  cont.nues  to  decrease  with  additional  truncation.  The  confidence  region  volume  initially 
decreases,  probably  because  of  the  high  variability  in  the  first  100  observations.  Additional 
deletions  after  the  first  100  data  points  result  in  increasing  the  confidence  region  volume.  The 
average  MMAE  truncation  point  is  indicated  with  double  horizontal  bars,  and  the  average 
truncation  point  selected  by  Schruben’s  algorithm  is  shown  with  a  single  bar. 
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Table  115.  Periodic  Truncation  Points  for  Open  Model  {Lq^(to)  =  26) 


no 

_ Ay _ 

Cov 

Vol 

■31 

0.50 

1.63 

8.10 

0 

0.71 

1.97 

9.15 

0.00 

± 

0.00 

0.85 

± 

0.03 

■a 

0.50 

1.64 

8.95 

0.89 

i 

0.02 

0.45 

± 

0.02 

Bi 

0.50 

1.63 

8.60 

0.88 

0.02 

0.47 

± 

0.02 

■a 

0.50 

1.63 

8.41 

0.87 

0.02 

0.51 

± 

0.02 

MM 

0.50 

1.63 

8.30 

0.86 

± 

0.02 

0.56 

± 

0.02 

0.50 

1.64 

8.23 

0.85 

± 

o 

o 

0.62 

± 

0.03 

Eg 

0.50 

1.64 

8.18 

0.84 

± 

0.02 

0.70 

± 

0.03 

700 

0.50 

1.64 

8.14 

0.83 

± 

0.02 

0.80 

± 

0.04 

800 

0.50 

1.64 

8.10 

0.82 

± 

0.02 

0.94 

± 

0.05 

900 

0.50 

1.64 

8.07 

0.81 

± 

0.02 

1.13 

± 

0.05 

1000 

0.50 

1.64 

8.07 

0.81 

± 

0.02 

1.40 

± 

0.06 

1100 

0.50 

1.64 

8.09 

0.81 

± 

0.02 

1.85 

± 

0.08 

1200 

0.50 

1.64 

8.10 

0.79 

± 

0.02 

2.57 

± 

0.11 

1300 

0.50 

1.64 

8.09 

0.77 

± 

0.02 

4.08 

± 

0.18 

1400 

0.50 

1.65 

8.08 

0.76 

± 

0.02 

8.24 

± 

0.41 

The  next  model  is  a  closed  network  of  servers.  Four  servers,  each  with  exponentially- 
distributed  service  times  with  means  of  0.8,  0.7,  0.6,  and  0.5,  are  arranged  in  a  ring.  Thirty 
customers,  initially  at  the  second  queue,  are  processed  around  the  ring.  The  system  is 
closed,  so  by  definition  no  additional  customers  arrive  and  no  current  customers  leave.  The 
responses  are  the  number  of  customers  in  line  or  being  served  for  the  first  three  servers.  (If 
all  four  queues  are  used,  the  multivariate  output  is  linearly  dependent.)  Ross  [79:330-335] 
provides  an  algorithm  to  calculate  the  analytic  steady-state  value  of  the  response.  For  this 
system,  the  vector  of  steady-state  responses  is  (19.33,  6.14,  2.89)^.  Beginning  with  all  the 
customers  queued  at  the  second  server  induces  a  significant  transient.  The  responses  are 
sampled  every  time  unit  until  1,500  observations  are  recorded. 

Table  116  shows  that,  for  the  1,000  Monte  Carlo  sets  of  runs,  the  mean  responses 
based  on  the  last  half  of  the  output  sequences  are  very  near  the  analytic  values.  Figures  26 
through  28  shows  the  average  estimated  covariance  matrix  and  autocovariance  matrices. 
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Table  116.  Open  Model  (4  Servers)  Queue  Lengths  Parameter  Estimation  Summary 


Model 

Closed  Model  (4  Servers) 

19.33 

19.31  0.65 

6.14 

6.16  0.54 

2.89  2.89  0.19 

Figure  26.  Average  Estimated  Covariance  Matrix  Syy  (dj,  ) 


4.114  (1.277) 
-2.892(1.055) 
-0.868(0.371) 


-2.892(1.055) 
3.087  (0.971) 
-0.142(0.264) 


-0.868  (0.371)  ■ 
-0.142  (0.264) 
1.025(0.228) 


Each  of  these  matrices  shows  a  negative  covariance  with  the  number  of  customers  at  other 
queues.  Since  the  number  of  customers  is  fixed,  this  negative  cross-correlation  is  logical. 

Figure  29  shows  the  average  state  transition  matrix  and  standard  deviations  over  the 
1,000  sets.  In  propagating  the  estimates  of  the  number  at  each  server,  the  previous  number 
at  that  server  has  the  largest  weight,  with  small  negative  weights  assigned  to  the  other 
server  queue  lengths.  The  average  Kalman  filter  gains  are  shown  in  Figure  30.  The  diagonal 
elements  are  relatively  close  to  1.0. 

Table  117  shows  the  results  of  applying  both  the  MMAE  and  Schruben’s  trucation 
point  selection  algorithms.  With  highly  interdependent  responses,  Schruben’s  algorithm 
selected  a  truncation  point  only  32  times  out  of  the  1,000  sets  of  runs.  In  contrast,  the 
MMAE  approach  selected  a  truncation  point  every  run.  Table  118  shows  that  the  coverage 


Figure  27.  Average  Estimated  Autocovariance  Matrix  at  One  Lag  ryy(l)  (dj,  ) 


(1) 


3.990(1.277) 
-2.833(1.054) 
-0.851  (0.371) 


-2.833(1.054) 
2.970  (0.970) 
-0.092  (0.264) 


-0.851  (0.371)  ■ 
-0.092  (0.264) 
0.918  (0.227) 
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Figure  28.  Average  Estimated  Autocovariaiice  Matrix  at  Two  Lags  Tyy(2)  ^  ^2)  ) 

■  3.868(1.275)  -2.773(1.053)  -0.821  (0.370)' 

-2.773(1.053)  2.865(0.969)  -0.056(0.263) 

-0.821  (0.370)  -0.056  (0.263)  0.832  (0.225) 


Figure  29.  Average  State  Transition  Matrix  $  (<7#r,c  ) 

■  0.892(0.028)  -0.082(0.027)  -0.072  (0.031)' 

-0.026(0.023)  0.936(0.025)  0  012  (0.027) 

-0.073  (0.025)  -0.062  (0.026)  0.825  (0.037) 


Figure  30.  Average  Kalman  Filter  Gain  K  ((7k,,c  ) 


■  0.877  (0.022) 
0.000  (0.000) 
0.000  (0.000) 


-0.097  (0.021) 
0.946  (0.019) 
0.000  (0.000) 


-0.098  (0.022) 
0.032  (0.019) 
0.796  (0.031) 


Table  117.  Truncation  Points  tTq 


- 1 

_ 1 

Fail 

Min 

Max 

Avg 

St  Dev 

Closed  Model  (4  Servers) 

MMAE  Algorithm 

0.000 

55 

344 

113.9 

36.5 

Schruben  Algorithm 

0.968 

70 

1460 

619.1 

430.9 
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Table  118.  Truncation  Point  Uq  Evaluations 


no  Cov  Vol 

Closed  Model  (4  Servers) 

MMAE  Algorithm 
Schruben  Algorithm 

113.9  ±  1.90 
619.1  ±  125.31 

0.88  ±  0.017  0.38  ±  0.010 
0.81  ±  0.114  1.64  ±  0.858 

Table  119.  Periodic  Output  Values  Points  for  the  Closed  Model  (4  Servers) 


n 

1919 

■31 

19.33 

6.14 

2.89 

1 

0.18 

0.43 

28.66 

1.19 

0.84 

1.04 

100 

16.73 

7.66 

7.89 

6.85 

3.50 

3.84 

19.14 

6.90 

6.29 

6.01 

2.93 

3.29 

BS 

19.27 

6.80 

6.22 

5.93 

2.88 

3.29 

400 

19.33 

6.74 

6.10 

5.80 

2.94 

500 

19.33 

6.85 

6.14 

5.91 

2.89 

3.25 

600 

19.38 

6.74 

5.79 

2.93 

3.36 

700 

19.27 

6.83 

6.16 

5.88 

2.89 

3.23 

800 

19.25 

6.79 

6.24 

5.83 

2.87 

3.24 

900 

19.25 

6.84 

6.16 

5.88 

2.92 

3.28 

1000 

19.29 

6.70 

6.15 

5.88 

2.94 

3.23 

1100 

19.45 

6.71 

5.78 

2.88 

3.23 

1200 

19.19 

6.81 

6.26 

2.89 

3.27 

1300 

19.36 

6.72 

6.16 

5.86 

2.83 

3.23 

1400 

19.29 

6.78 

6.18 

5.88 

2.88 

3.24 

1500 

19.25 

6.85 

6.16 

5.89 

2.93 

3.33 

rate  for  the  sequences  truncated  at  the  MMAE  selected  point  is  very  near  nominal.  The 
coverage  rate  and  volume  for  Schruben’s  algorithm  are  based  only  on  the  sequences  resulting 
from  the  32  selected  truncation  points. 

Table  119  shows  periodic  output  responses,  and  Table  120  shows  the  estimated  re¬ 
sponses,  coverage  rates,  and  confiden  :e  region  volume  for  periodic  truncation  points.  These 
two  tables  indicate  that  the  MMAE  average  selected  truncation  point  of  113.9  is  both  ef¬ 
fective  in  terms  of  removing  the  transient  and  efficient  by  not  excessively  truncating  the 
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Table  120.  Periodic  Truncation  Points  for  the  Closed  Model  (4  Servers) 


fio 

My 

Cov 

Vol 

My 

19.33  6.14  2.89 

0 

18.62  6.73  2.97 

0.78  ±  0.02 

0.36  ±  0.01 

100 

19.24  6.21  2.91 

0.90  ±  0.02 

0.38  ±  0.01 

200 

19.31  6.16  2.89 

0.88  ±  0.02 

0.42  ±  0.01 

300 

19.31  6.16  2.89 

0.88  ±  0.02 

0.46  ±  0.01 

400 

19.31  6.16  2.89 

0.87  ±  0.02 

0.52  ±  0.01 

500 

19.32  6.16  2.89 

0.88  ±  0.02 

0.60  ±  0.02 

600 

19.32  6.16  2.89 

0.88  ±  0.02 

0.70  ±  0.02 

700 

19.31  6.16  2.89 

0.88  ±  0.02 

0.83  ±  0.02 

800 

19.31  6.16  2.89 

0.89  ±  0.02 

0.99  ±  0.03 

900 

19.32  6.16  2.89 

0.87  ±  0.02 

1.24  ±  0.04 

1000 

19.32  6.16  2.88 

0.89  ±  0.02 

1.62  ±  0.05 

1100 

19.31  6.17  2.88 

0.86  ±  0.02 

2.21  ±  0.07 

1200 

19.31  6.17  2.88 

0.87  ±  0.02 

3.26  ±  0.11 

1300 

19.30  6.18  2.89 

0.83  ±  0.02 

5.69  ±  0.21 

1400 

19.26  6.21  2.90 

0.81  ±  0.02 

13.11  ±  0.57 

sequences. 

The  closed  model  is  also  run  with  five  servers.  The  mean  service  times  are  0.8,  0.7, 
0.6,  0.5  and  0.4  time  units.  The  analytic  numbers  of  entities  at  each  of  the  first  four  servers 
are  (18.46,  6.05,  2.88,  1.63)^.  Table  121  demonstrates  that  these  values  are  estimated  fairly 
closely  with  the  last  half  of  the  output  sequences.  Figures  31  through  33  show  the  positive 
autocovariances  and  the  negative  covariances  among  the  responses.  Figures  34  and  35  show 
the  average  state  transition  matrix  and  average  Kalman  filter  gain.  Since  the  off-diagonal 
elements  in  the  state  transition  matrix  and  Kalman  filter  gains  are  small  on  the  average,  the 
Kalman  filter  relies  mostly  on  measurements  of  each  response  for  estimating  the  correspond¬ 
ing  state. 

Statistics  on  the  truncation  points  are  shown  in  Table  122  for  both  the  MMAE  and 
Schruber’s  algorithms.  Schruben’s  algorithm  selected  a  truncation  point  in  only  3  of  the 
1,000  sets  of  runs.  Schruben’s  approach  appears  not  to  work  well  with  negatively  correlated 
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Table  121.  Closed  Model  (5  Servers)  Parameter  Estimation  Summary 


Model 

2/2 

Avj 

Mvs  2/3 

Aw  ^Vi 

tJ-VK  2/4 

Av4  ^V4 

Closed  Model 

18.46  18.48  0.63 

6.05  6.03  0.52 

2.88  2.88  0.19 

1.63  1.63  0.08 

Figure  31.  Average  Estimated  Covariance  Matrix  Eyy  (dj,  ) 


4.093  (1.262) 
-2.732(1.000) 
-0.851  (0.375) 
-0.350  (0.147) 


-2.732  (1.000) 
2.966  (0.899) 
-0.157  (0.258) 
-0.055(0.115) 


-0.851  (0.375) 
-0.157  (0.258) 
1.026  (0.235) 
-0.012  (0.063) 


-0.350  (0.147)  ■ 
-0.055  (0.115) 
-0.012  (0.063) 
0.419  (0.068) 


Figure  32.  Average  Estimated  Autocovariance  Matrix  at  One  Lag  ryy(l)  ) 


3.969(1.262) 
-2.673  (0.999) 
-0.843  (0.374) 
-0.332  (0.147) 


-2.673  (0.999) 
2.850  (0.898) 
-0.107  (0.257) 
-0.048  (0.115) 


-0.843  (0.374) 
-0.107(0.257) 
0.919  (0.234) 
0.028  (0.062) 


-0.332  (0.147)  ■ 
-0.048  (0.115) 
0.028  (0.062) 
0.324  (0.066)  . 


Figure  33.  Average  Estimated  Autocovariance  Matrix  at  Two  Lags 


ryy(2)  (dj 


=  (2) 


3.846  (1.260) 
-2.618  (0.999) 
-0.822  (0.373) 
-0.305  (0.146) 


-2.618(0.999) 
2.745  (0.897) 
-0.072  (0.256) 
-0.038  (0.114) 


-0.822  (0.373) 
-0.072  (U.256) 
0.834  (0.231) 
0.048  (0.062) 


-0.305(0.146)  ■ 
-0.038  (0.114) 
0.048  (0.062) 
0.261  (0.064)  . 
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Figure  34.  Average  State  Transition  Matrix  $  ) 


■  0.823  (0.056) 

-0.020  (0.045) 
-0.075  (0.046) 
.  -0.112  (0.045) 


-0.151  (0.055) 
0.941  (0.046) 
-0.063  (0.046) 
-0.110  (0.046) 


-0.151  (0.056) 
0.016  (0.047) 
0.823  (0.053) 
-0.082  (0.046) 


-0.101  (0.068)  ■ 
0.003  (0.056J 
-0.007  (0.058) 
0.670  (0.070)  . 


Figure  35.  Average  Kalman  Filter  Gain  K  (^Kr,e  ) 


0.784  (0.029) 
0.000  (0.000) 
0.000  (0.000) 
0.000  (0.000) 


0.190  (0.029) 
0.955  (0.024) 
0.000  (0.000) 
0.000  (0.000) 


-0.165  (0.031) 
0.008  (0.026) 
0.052  (0.030) 
0.614(0.041) 


-0.200  (0.029) 
0.041  (0.024) 
0.846  (0.033) 
0.000  (0.000) 


Table  122.  Truncation  Points  Uq 


J 

Fail 

Min 

Max 

Avg 

St  Dev 

Closed  Model  (5  Servers) 

MMAE  Algorith;i-. 

0.000 

41 

301 

105.5 

32.7 

Schruben  Algorithm 

0.997 

500 

1290 

893.3 

322.5 
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Table  123.  Truncation  Point  TOq  Evaluations 


no  Cov  Vol 

Closed  Model  (5  Servers) 

MMAE  Algorithm 
Schruben  Algorithm 

105.5  ±  1.70 
893.3  ±  543.73 

0.85  ±  0.018  0.07  ±  0.002 
0.33  ±  0.795  1.25  ±  2.581 

Table  124.  Periodic  Output  Values  Points  for  Closed  Model  (5  Servers) 


n 

1^9 

^99 

wssa 

■31 

18.46 

6.05 

2.88 

1.63 

r 

0.08 

0.29 

28.59 

1.19 

0.85 

1.03 

0.33 

0.59 

100 

16.40 

7.48 

7.43 

6.54 

3.29 

3.68 

1.79 

2.25 

200 

18.35 

6.84 

6.15 

5.87 

2.87 

3.21 

1.64 

2.04 

300 

18.28 

6.82 

6.14 

5.80 

2.94 

3.37 

1.67 

2.16 

400 

18.36 

6.80 

6.14 

5.77 

2.89 

3.25 

1.63 

2.00 

500 

18.30 

6.81 

6.11 

5.78 

2.96 

3.38 

1.66 

2.10 

600 

18.45 

6.79 

6.15 

5.76 

2.79 

3.23 

1.59 

2.04 

700 

18.47 

6.74 

5.99 

5.68 

1.63 

2.06 

800 

18.41 

6.78 

6.10 

5.77 

2.86 

1.64 

2.07 

900 

18.47 

6.64 

2.86 

1.64 

2.05 

1000 

18.42 

6.74 

6.14 

5.76 

2.88 

3.23 

1.59 

2.03 

1100 

18.53 

6.70 

5.98 

5.64 

2.87 

3.28 

1.63 

2.09 

1200 

18.44 

6.75 

5.69 

3.22 

1.68 

2.08 

1300 

18.45 

6.69 

5.68 

2.88 

3.23 

1.65 

2.07 

1400 

18.47 

6.74 

2.92 

3.32 

1.63 

2.06 

1500 

18.44 

6.70 

5.97 

5.66 

2.93 

3.31 

1 

1.65 

2.07 

output  responses.  The  MMAE  algorithm  truncated  105.5  observations  on  the  average.  The 
MMAE  truncated  sequences  result  in  a  coverage  rate  of  85  percent  and  an  average  confidence 
region  volume  of  0.07.  Only  one  of  three  truncated  points  selected  by  Schruben’s  routine 
result  in  a  confidence  region  which  covered  the  analytic  mean. 

The  table  of  periodic  output  values,  Table  124,  shows  that  large  transient  effects  last 
less  than  200  observations,  but  small  transient  effects  are  seen  until  about  the  600th  obser¬ 
vation.  From  Table  125,  the  transient  is  insignificant  on  the  confidence  region  coverage  rate 
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Table  125.  Periodic  Truncation  Points  for  Closed  Model  (5  Servers ) 


no 

_  ibL _ 

Cov 

Vol 

m 

18.46 

6.05 

2.88 

1.63 

0 

17.80 

6.58 

2.95 

1.65 

0.78 

± 

ESS 

± 

0.00 

100 

18.38 

6.10 

2.89 

1.64 

0.88 

± 

nBa 

Hi 

± 

0.00 

200 

18.43 

6.06 

2.88 

1.63 

0.89 

± 

0.02 

0.08 

± 

0.00 

300 

18.44 

6.06 

2.88 

1.63 

0.87 

± 

0.02 

0.09 

± 

0.00 

400 

18.45 

6.05 

2.88 

1.63 

0.87 

± 

0.02 

0.10 

± 

0.00 

500 

18.46 

6.04 

2.88 

1.63 

0.88 

± 

0.02 

0.12 

± 

0.00 

600 

18.47 

6.03 

2.88 

1.63 

0.89 

± 

0.02 

0.15 

± 

0.01 

700 

18.48 

6.03 

2.88 

1.63 

0.89 

± 

0.02 

0.19 

± 

0.01 

800 

18.48 

6.03 

2.87 

1.63 

0.90 

± 

0.02 

0.25 

± 

0.01 

900 

18.48 

6.03 

2.88 

1.63 

0.88 

± 

0.02 

0.33 

± 

0.01 

1000 

18.48 

6.02 

2.88 

1.63 

0.86 

± 

0.02 

0.47 

± 

0.02 

1100 

18.49 

6.02 

2.87 

1.63 

0.87 

± 

0.02 

0.72 

± 

0.03 

1200 

18.48 

6.03 

2.87 

1.63 

0.85 

± 

0.02 

1.23 

± 

0.05 

1300 

18.49 

6.02 

2.88 

1.63 

0.85 

± 

0.02 

2.60 

± 

0.11 

1400 

18.48 

6.01 

2.88 

1.64 

0.81 

i 

0.02 

8.31 

± 

0.41 

after  truncating  100  observations. 

The  final  output  considered  is  the  waiting  and  service  times  for  entities  in  a  simulation 
of  two  tandem  M/M/1  queues.  Both  the  interarrival  and  service  times  are  exponentially 
distributee.  The  mean  interarrival  time  is  1.0  time  unit,  and  the  means  of  the  service  times 
are  0.6  and  0.8,  respectively.  Ross  [79]  shows  that  the  queues  are  independent  with  the 
analytic  solutions  the  same  as  individual  M/M/1  queues.  The  first  response  is  the  waiting 
time  plus  service  time  at  the  first  server,  and  the  second  response  is  total  time  in  the  system. 
Since  the  second  response  is  the  sum  the  first  response  plus  an  independent  random  variable, 
the  responses  are  moderately  correlated. 

This  output  sequence  is  an  example  of  “statistics  based  on  observations” ,  whereas  the 
previous  sequences  are  “statistics  based  on  time-persistent  variables” .  In  this  case,  the  state 
transition  matrix  represents  the  relationship  between  subsequent  entities  rather  than  the 
relationship  over  a  fixed  time  interval.  The  state  transition  is  estimated  in  the  same  way 
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Table  ,26.  Open  Model  Wciiting  Times  Parameter 


Model 

Open  Model  Waiting  Times 

1.50  1.50  0.07 

5.50  5.49  0.44 

istimation  Summary 


Figure  36.  Average  Estimated  Covariance  Matrix  Eyy  ^  ) 

■  0.223  (0.045)  0.216  (0.090)  ' 

0.216  (0.090)  1.659  (0.654) 


eind  is  still  held  constant  in  the  Kalman  filter. 

The  steady-state  models  are  estimated  using  the  last  half  of  the  output  sequences. 
Table  126  shows  that  the  average  steady-state  mean  estimates  are  centered  on  the  analytic 
means.  Figures  36  through  38  indicate  that  the  output  sequences  have  positive  covariances 
and  positive  autocovariances. 

Figure  39  shows  the  average  state  transition  matrix,  and  Figure  40  shows  the  average 
Kalman  filter  gains.  Since  the  off-diagonal  elements  are  near  zero,  the  MMAE  approach  does 
not  rely  on  the  covariance  between  the  responses. 

The  MMAE  algorithm  never  failed  to  select  a  truncation  point,  while  Schruben’s  ap¬ 
proach  failed  in  4.3  percent  of  the  tests,  as  shown  in  Table  127.  The  MMAE  truncation 
points  are  considerably  earlier  in  the  sequence,  with  much  less  variance  than  the  trunca¬ 
tion  point"^  .‘elected  by  Schruben’s  algorithm.  Table  128  shows  that  the  resulting  coverage 


Figure  37.  Average  Estimated  Autocovariance  Matrix  at  One  Lag  fyy(l)  (dj,  ) 


0.187(0.045)  0.192(0.090) 
0.192  (0.090)  1.584  (0.654) 
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Figure  38.  Average  Estimated  Autocovariance  Matrix  at  Two  Lags  fyy(2)  ,^(2)  ) 


0.160  (0.044)  0.174  (0.090) 
0.174  (0.090)  1.517  (0.653) 


Figure  39.  Average  State  Transition  Matrix  $  ) 

0.838  (0.034)  0.007  (0.008)  ' 

-0.066  (0.031)  0.959  (0.017) 


Figure  40.  Average  Kalman  Filter  Gain  K  (dK,,.  ) 

'  0.822  (0.032)  0.008  (0.009)  ' 

0.000  (0.000)  0.958  (0.016) 


Table  127.  Truncation  Points  Up 


Fail 

Min  Mcix 

Avg  St  Dev 

Open  Model  Waiting  Times 

MMAE  Algorithm 
Schruben  Algorithm 

0.000 

0.043 

9  310 

0  1480 

36.1  24.2 

236.3  283.3 
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Table  128.  Truncation  Point  np  Evaluations 


no 

Cov  Vol 

0] 

aen  Model  Waiting  Times 

MMAE  Algorithm 
Schruben  Algorithm 

36.1  ±  1.26 
236.3  ±  15.06 

0.87  ±  0.017  0.26  ±  0.007 
0.88  ±  0.018  0.43  ±  0.057 

Table  129.  Periodic  Output  Values  Points  for  Open  Model  Waiting  Times 


n 

Avn.l 

Avn.J 

1.50 

5.50 

1 

0.62 

0.61 

1.43 

1.03 

100 

1.49 

1.49 

5.31 

4.00 

200 

1.51 

1.53 

5.47 

4.16 

300 

1.50 

1.49 

5.48 

4.19 

400 

1.49 

1.49 

5.51 

4.31 

500 

1.50 

1.48 

5.50 

4.27 

600 

1.50 

1.50 

5.53 

4.28 

700 

1.47 

1.48 

5.53 

4.37 

800 

1.51 

1.52 

5.54 

4.31 

900 

1.50 

1.49 

5.52 

4.33 

1000 

1.49 

1.49 

5.43 

4.22 

1100 

1.50 

1.52 

5.49 

4.27 

1200 

1.50 

1.51 

5.54 

4.31 

1300 

1.50 

1.50 

5.57 

4.42 

1400 

1.49 

1.50 

5.48 

4.25 

1500 

1.50 

1.49 

5.50 

4.21 

rates  are  nearly  equal,  but  that  the  MMAE  truncation  point  result  in  considerably  smaller 
confidence  region  volumes. 

Table  129  shows  periodic  responses.  The  analytic  means  are  obtained  at  the  400th 
observation  on  average.  The  evaluation  of  periodic  truncation  points  are  given  in  Table  130. 
Very  near  nominal  coverage  rates  are  achieved  with  only  small  amounts  of  truncation.  The 
MMAE  approach  which  truncates  early  in  the  sequence  appears  preferable  to  Schruben’s 
algorithm. 
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7. 6  Summary 

The  multivariate  MMAE  truncation-point  selection  algorithm  is  very  successful  on  the 
output  of  four  simulation  models.  The  truncated  sequences  have  very  near  nominal  coverage 
rates  ajid  relatively  small  confidence  region  volumes.  The  test  cases  include  two  cases  of 
“statistics  based  on  time-persistent  variables”:  the  open  queuing  system  with  uncorrelated 
responses  and  the  closed  queuing  system  with  negatively  correlated  responses.  The  third  test 
case  is  an  example  of  “statistics  based  upon  observations”  from  an  open  queuing  system. 
In  all  of  these  cases,  the  sequences  truncated  with  the  MMAE  points  are  more  effective 
(in  terms  of  coverage  rates)  and  more  efficient  (in  terms  of  confidence  region  volume)  than 
sequences  truncated  with  points  selected  by  Schruben’s  [86]  algorithm. 

Schruben’s  method  [86]  rarely  selected  a  truncation  point  for  the  closed  queuing  system. 
In  comparision  to  the  other  Ceises,  these  responses  have  relatively  stronger  correlation  and 
further  have  negative  correlation.  Schruben’s  method  is  based  on  the  sample  mean  vector 
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and  sample  covariance  matrix  estimated  with  only  the  last  batch  from  each  observation. 
Since  the  recommended  batches  are  small,  only  5  to  10  observations,  very  little  data  is 
used  for  these  estimates.  The  variance  of  these  estimates  may  have  contributed  to  the  poor 
performance  of  his  technique.  In  his  article,  Schruben  [86]  presents  only  one  trial  case  for 
his  algorithm.  His  example  case  is  based  on  the  time-sharing  computer  model  studied  by 
Adirir  and  Avi-Itzhak  [3].  The  responses  are  the  computer  response  time,  a  “statistic  based 
on  observations”  and  a  zero-one  indicator  variable  of  the  cpu’s  busy/idle  status,  which  is 
a  “statistic  based  on  time”.  Because  the  sequences  are  different  types,  how  the  data  is 
collected  in  pairs  is  not  obvious.  Although  not  specified,  it  appears  Schruben  took  these  two 
seperate  univariate  sequences  and  matched  up  corresponding  terms  to  form  his  multivaritate 
sequence.  Since  the  pairs  of  responses  may  not  have  measured  the  system  under  the  same 
conditions,  the  mulitvariate  sequence  may  not  have  significant  covaricince.  Schruben  may 
have  tested  his  multivariate  algorithm  under  the  conditions  of  little  covariance.  As  seen 
in  this  chapter,  his  algorithm  performs  well  with  no  covariance,  but  less  satisfactorily  with 
negative  covariance. 
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VIII.  Summary  and  Recommendations 


8. 1  Summary 

The  significcince  of  this  research  is  the  introduction  of  new  concepts  to  the  field  of 
discrete-event  simulation  output  analysis.  Specifically,  modeling  with  measurement  noise, 
applying  Kalman  filters,  and  using  Multiple  Model  Adaptive  Estimation  (MMAE)  are  all 
new  approaches  to  analyzing  simulation  output.  As  a  result  of  applying  these  concepts, 
new  algorithms  for  truncation-point  selection  for  both  univariate  and  multivariate  output 
sequences  are  proposed  and  tested.  These  algorithms  are  applicable  to  both  “statistics 
based  on  time-persisitent  variables '  and  “statistics  based  on  observations” .  The  truncation- 
point  selection  algorithms  based  on  MMAE  result  in  truncated  sequences  with  better  mean 
estimates  and  confidence  intervals. 

This  research  begins  with  a  review  of  simulation  output  analysis.  Particular  focus  is 
placed  on  the  initial  data  bias  or  warm-up  problem.  An  overview  of  stochastic  processes  is 
presented,  leading  to  an  introduction  of  the  Kalman  filter.  The  Kalman  filter  is  a  state-space 
stochastic  estimation  algorithm.  Several  approaches  for  system  identification  and  estimation 
are  reviewed. 

Using  the  foundation  of  previous  simulation  research  and  stochastic  estimation  with 
the  Kalmein  filter,  two  methods  to  select  truncation  points  to  eliminate  the  initial  data  bias 
are  developed.  The  first  truncation-point  selection  approach  is  based  on  residual  monitoring 
using  a  single  Kalman  filter.  The  knowledge  gained  from  this  attempt  laid  the  foundation 
for  an  improved  approach  based  on  Multiple  Model  Adaptive  Estimation  (MMAE).  MMAE 
is  an  estimation  technique  that  uses  a  bank  of  Kalman  filters  with  different  estimates.  In 
the  MMAE  method,  the  output  is  estimated  as  an  autoregressive  model  observed  with  mea¬ 
surement  noise  using  the  last  half  of  the  output  sequence.  The  MMAE  filters  have  different 
estimates  for  the  output  means.  Based  on  the  filters’  residuals,  a  time- varying  MMAE  mean 
estimate  is  calculated.  When  the  MMAE  mean  estimate  is  close  to  the  mean  estimator  from 
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the  last  half  of  the  output  sequence  for  a  predetermined  number  of  observations,  a  truncation 
point  is  selected. 

The  MMAE  truncation  algorithm  performs  successfully  on  a  single  long  run  of  simu¬ 
lation  output.  However,  for  single  long  runs,  the  effects  of  the  initial  transient  on  making 
confidence  intervals  may  be  negligible.  The  MMAE  truncation  algorithm  is  very  successful 
with  multiple  replications  for  both  univariate  and  multivariate  output  sequences.  The  se¬ 
lected  truncation  points  effectively  reduce  the  effects  of  the  initial  bias,  without  excessive 
deletion  which  increases  the  variance  of  the  estimates. 

8.2  Recommendations  for  Improvements  to  the  MMAE  Truncation- Point  Selection  Algo¬ 
rithms 

Instead  of  the  correlation  technique,  perhaps  a  different  estimation  approach  for  the 
steady-state  parameters  could  be  employed.  Law  and  Kelton  [53:286]  show  that  autocor¬ 
relation  estimates  may  be  biased.  1'  the  sample  size  is  fixed  to  the  last  half  of  the  output 
sequence,  MLE  or  least  squares  can  be  used  instead  of  the  correlation  technique.  More  elab¬ 
orate  estimation  schemes  may  be  possible.  For  example.  Minimum  Distance  Estimation  has 
been  found  to  outperform  MLE  for  small  static  samples  [25].  If  the  estimate  of  k  can  be 
improved,  the  variance  estimates  may  be  more  reliable  and  may  prove  useful. 

A  better  scheme  for  determining  the  MMAE  filter  spacing  or  truncation  point  may 
be  possible.  The  tested  algorithm  estimates  the  steady-state  model  and  finds  the  average, 
minimum,  and  maximum  for  each  response  from  the  last  half  of  the  output  sequences.  The 
MMAE  filters  are  positioned  at  each  combination  of  the  response  averages,  minimum,  and 
maximum.  The  truncation  point  is  selected  when  the  MMAE  time- varying  mean  estimate  is 
within  one  of  the  Euclidean  distance  to  the  response  averages  for  5  consecutive  observations. 
These  results  are  based  on  an  initial  approach  without  attempting  to  optimize.  Perhaps  a 
different  vector  norm  or  consecutive  number  of  observations  may  improve  the  results.  At 
a  minimum,  a  theshold  for  the  difference  in  mean  estimates  in  terms  of  the  sample  output 
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mean,  rather  than  the  absolute  value  of  one,  may  make  the  algorithm  applicable  to  a  wider 
range  of  model  outputs. 

A  transformation  of  the  data,  such  as  natural  logarithms  of  each  observation,  may 
reduce  the  effect  of  the  exponential  output  distribution  from  many  queuing  systems.  If  the 
output  distribution  is  closer  to  the  normal  distribution,  the  Kalman  filter  assumptions  of 
normally  distributed  noise  terms  may  be  more  appropriate. 

The  underlying  model  could  possibly  be  changed  from  an  AR(2)  with  measurement 
noise  model  to  regulated  Brownian  motion.  Several  researchers  [1,  30,  102]  have  applied 
regulated  Brownian  motion  models  to  simulation  output  with  success.  The  resulting  Kalman 
filter  may  have  significant  nonlinearities. 

8.S  Potential  Extensions  to  Other  Discrete- Event  Simulation  Analysis  Applications 

Several  other  potential  applications  of  Kalman  filters  for  simulation  output  analysis, 
which  not  developed  in  this  dissertation,  are  presented.  These  proposals  are  grouped  into 
four  categories:  mean  vector  estimation,  covariance  estimation,  variance  reduction,  and  run 
length  control.  The  univariate  case  is  a  special  case  of  each  multivariate  proposal. 

8.3.1  Multivariate  Estimation  of  the  Mean  Vector.  Develop  an  improved  method  of 
multivariate  estimation  of  the  mean  vector  with  a  joint  confidence  region  for  the  estimated 
parameters.  Three  possibilities  are  apparent,  but  each  of  these  concepts  is  extremely  depen¬ 
dent  upon  the  definition  of  the  system  states  used  in  the  Kalman  filter. 

The  first  idea  for  developing  a  confidence  iri.,erval  for  the  parameters  is  to  relate  the 
Kalman  filter  model  estimates  and  covariance  matrices  to  the  estimated  parameter’s  co- 
variance  matrix.  In  the  process  of  running  the  Kalman  filter,  a  covariance  matrix  of  the 
Kalman  filter  states  is  developed.  Along  with  this  covariance  matrix  of  states,  the  cross¬ 
covariance  between  the  states  and  the  parameters  could  be  explored  if  nonlinear  Kalman 
filter  approaches  (See  Maybeck  [58])  are  applied.  If  this  information  could  be  related  to  the 
covariance  matrix  of  the  parameter  estimates  of  the  simulation  output  distributions,  a  new 
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technique  for  developing  confidence  intervals  and  regions  can  be  tested.  Another  potential 
source  of  variance  estimates  are  the  probabilities  from  Multiple  Model  Adaptive  Estimation 
[58:68-144].  These  approaches  are  tested  in  the  univariate  case  by  Howard  [37]  and  Howard, 
Gallagher,  Bauer  and  Maybeck  [39,  38]. 

The  second  concept  addresses  the  problem  of  asynchronous  output.  Asynchronous  out¬ 
put  occurs  when  each  output  vector  has  only  some  of  the  responses  [12,  90,  91].  If  functions 
of  the  simulation  output  responses  are  included  as  Kalman  filter  states,  the  information  from 
the  asynchronous  output  could  be  incorporated  using  scalar  measurement  update  techniques. 
(For  scalar  measurement  updates,  see  Maybeck  [57].)  The  Kalman  filter  algorithm  may  pro¬ 
duce  the  estimated  mean  and  covariance  matrix  of  interest  for  the  multivariate  stochastic 
process.  The  difficulty  is  to  define  and  estimate  the  state  transition  matrix  $.  Synder  [95] 
may  provide  the  necessary  mathematical  foundations. 

Third,  mean  estimates  are  calculated  currently  by  a  weighted  average  of  observation 
values,  but  the  Kalman  filter  produces  a  sequence  of  state  estimates.  Perhaps,  the  Kalman 
filter  state  estimates  may  provide  an  improved  estimate  of  the  mean  vector.  Howard  [37] 
finds  there  is  very  little  difference  between  the  average  state  estimate  and  the  average  of 
the  data.  This  occurs  for  his  applications  because  the  Kalman  filter  gains  are  almost  1.0, 
which  simply  makes  each  state  estimate,  the  propagation  of  the  last  two  data  values.  With 
smaller  Kalman  filter  gains,  the  state  estimates  may  smooth  the  sequence  by  decreasing  the 
magnitude  of  outliers. 

8.3.2  Covariance  Matrix  Estimation.  Besides  estimating  the  variance  of  mean  esti¬ 
mates,  the  covariance  of  the  underlying  multivariate  stochastic  process  is  also  of  interest.  In 
fitting  the  Kalman  filter  to  the  simulation  output,  some  estimate  of  the  covariance  of  the 
underlying  stochastic  process  is  embedded  in  the  analysis  process.  The  MMAE  technique 
[58:68-144]  can  also  be  used  to  estimate  the  covariance  matrix.  In  the  univariate  case,  this 
variance  estimate  can  be  compared  with  the  Welch’s  [101:300-302]  jackknife  technique  to 
estimate  the  system  variance. 
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8.S.S  Improved  Control  Variates  Applications.  Variance  reduction  is  a  collection  of 
techniques  to  reduce  the  variance  of  estimates.  One  specific  class  of  variance  reduction 
techniques  is  the  method  of  control  variates.  Control  variate  methods  use  the  covariance 
between  the  input  pseudorandom  control  variates  and  the  output  sequence.  Based  on  this 
covariance,  conditional  estimates  of  the  parameters  of  interest  are  determined,  given  the 
actual  realizations  of  the  control  variates.  While  the  current  techniques  use  the  aggregate 
average  of  the  control  variates  over  an  entire  simulation  run,  the  Kalman  filter  potentially  can 
be  used  to  develop  the  covariance  based  on  the  individual  control  variates  and  the  affected 
entities.  Since  more  information  is  used,  better  estimates  with  smaller  variances  may  result. 

8. 3-4  Enhanced  Simulation  Run-Length  Control.  One  difficulty  with  simulation  out¬ 
put  analysis  is  to  determine  the  necessary  run  length  to  achieve  a  predetermined  accuracy  in 
the  parameter  estimates.  Any  information  gained  from  the  Kalman  filter  about  the  correla¬ 
tion  among  the  output  sequence  or  the  stocheistic  process  variance  can  be  used  to  improve  the 
current  run  length  determination  techniques.  Perhaps  a  “rule  of  thumb”  relating  necessary 
simulation  run  length  from  the  estimate  of  can  be  developed. 
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